Oct 31 03:14:53.174015 ip-10-0-142-76 systemd[1]: Starting Kubernetes Kubelet... Oct 31 03:14:53.549340 ip-10-0-142-76 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:14:53.549340 ip-10-0-142-76 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 03:14:53.549340 ip-10-0-142-76 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:14:53.549340 ip-10-0-142-76 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:14:53.554820 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554732 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:14:53.554820 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554819 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:14:53.554820 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554824 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:14:53.554918 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554828 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:14:53.554918 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554831 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:14:53.554918 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554836 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:14:53.554918 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554841 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:14:53.554918 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554845 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:14:53.554918 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554848 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:14:53.554918 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554853 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:14:53.554918 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554857 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:14:53.554918 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554860 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:14:53.554918 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554863 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:14:53.554918 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554872 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:14:53.554918 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554875 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:14:53.554918 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554878 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:14:53.554918 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554881 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:14:53.554918 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554884 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:14:53.554918 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554886 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:14:53.554918 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554889 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:14:53.554918 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554892 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:14:53.555335 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554895 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:14:53.555335 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554897 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:14:53.555335 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554901 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:14:53.555335 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554904 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:14:53.555335 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554906 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:14:53.555335 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554909 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:14:53.555335 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554912 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:14:53.555335 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554915 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:14:53.555335 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554917 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:14:53.555335 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554920 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:14:53.555335 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554923 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:14:53.555335 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554926 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:14:53.555335 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554929 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:14:53.555335 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554932 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:14:53.555335 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554935 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:14:53.555335 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554938 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:14:53.555335 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554941 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:14:53.555335 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554944 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:14:53.555335 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554947 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:14:53.555335 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554950 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:14:53.555809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554952 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:14:53.555809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554955 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:14:53.555809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554958 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:14:53.555809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554960 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:14:53.555809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554964 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:14:53.555809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554967 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:14:53.555809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554969 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:14:53.555809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554972 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:14:53.555809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554974 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:14:53.555809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554977 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:14:53.555809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554980 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:14:53.555809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554982 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:14:53.555809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554986 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:14:53.555809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554989 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:14:53.555809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554992 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:14:53.555809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554995 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:14:53.555809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.554998 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:14:53.555809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555000 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:14:53.555809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555004 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:14:53.555809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555006 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:14:53.556272 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555009 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:14:53.556272 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555012 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:14:53.556272 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555014 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:14:53.556272 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555017 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:14:53.556272 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555020 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:14:53.556272 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555022 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:14:53.556272 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555025 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:14:53.556272 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555028 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:14:53.556272 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555030 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:14:53.556272 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555033 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:14:53.556272 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555036 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:14:53.556272 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555039 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:14:53.556272 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555042 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:14:53.556272 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555045 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:14:53.556272 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555047 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:14:53.556272 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555050 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:14:53.556272 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555053 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:14:53.556272 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555055 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:14:53.556272 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555059 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:14:53.556272 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555062 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:14:53.556772 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555064 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:14:53.556772 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555067 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:14:53.556772 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555069 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:14:53.556772 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555072 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:14:53.556772 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555076 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:14:53.556772 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555078 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:14:53.556772 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555081 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:14:53.556772 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555084 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:14:53.556772 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555087 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:14:53.556772 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555089 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:14:53.556772 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555092 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:14:53.556772 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555095 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:14:53.556772 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555097 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:14:53.556772 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555100 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:14:53.556772 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555103 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:14:53.556772 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555105 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:14:53.556772 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555108 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:14:53.556772 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555110 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:14:53.556772 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555113 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:14:53.556772 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555116 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:14:53.557513 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555119 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:14:53.557513 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555121 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:14:53.557513 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555124 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:14:53.557513 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555126 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:14:53.557513 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555130 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:14:53.557513 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555938 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:14:53.557513 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555946 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:14:53.557513 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555949 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:14:53.557513 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555951 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:14:53.557513 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555955 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:14:53.557513 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555958 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:14:53.557513 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555961 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:14:53.557513 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555963 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:14:53.557513 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555966 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:14:53.557513 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555969 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:14:53.557513 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555972 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:14:53.557513 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555975 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:14:53.557513 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555978 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:14:53.557513 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555981 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:14:53.557513 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555984 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:14:53.558254 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555986 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:14:53.558254 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555990 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:14:53.558254 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555992 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:14:53.558254 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555995 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:14:53.558254 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.555998 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:14:53.558254 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556001 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:14:53.558254 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556004 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:14:53.558254 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556006 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:14:53.558254 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556009 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:14:53.558254 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556012 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:14:53.558254 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556015 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:14:53.558254 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556018 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:14:53.558254 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556021 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:14:53.558254 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556024 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:14:53.558254 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556026 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:14:53.558254 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556029 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:14:53.558254 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556031 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:14:53.558254 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556035 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:14:53.558254 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556037 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:14:53.558734 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556040 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:14:53.558734 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556043 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:14:53.558734 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556045 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:14:53.558734 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556048 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:14:53.558734 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556050 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:14:53.558734 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556053 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:14:53.558734 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556056 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:14:53.558734 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556058 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:14:53.558734 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556061 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:14:53.558734 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556064 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:14:53.558734 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556067 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:14:53.558734 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556070 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:14:53.558734 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556073 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:14:53.558734 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556075 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:14:53.558734 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556078 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:14:53.558734 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556080 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:14:53.558734 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556083 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:14:53.558734 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556086 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:14:53.558734 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556088 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:14:53.558734 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556091 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:14:53.559218 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556094 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:14:53.559218 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556096 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:14:53.559218 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556099 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:14:53.559218 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556102 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:14:53.559218 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556104 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:14:53.559218 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556107 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:14:53.559218 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556110 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:14:53.559218 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556113 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:14:53.559218 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556115 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:14:53.559218 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556118 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:14:53.559218 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556120 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:14:53.559218 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556123 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:14:53.559218 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556126 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:14:53.559218 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556129 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:14:53.559218 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556131 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:14:53.559218 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556134 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:14:53.559218 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556138 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:14:53.559218 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556143 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:14:53.559218 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556146 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:14:53.559670 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556150 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:14:53.559670 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556152 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:14:53.559670 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556157 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:14:53.559670 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556161 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:14:53.559670 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556164 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:14:53.559670 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556167 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:14:53.559670 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556170 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:14:53.559670 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556173 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:14:53.559670 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556175 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:14:53.559670 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556178 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:14:53.559670 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556181 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:14:53.559670 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556183 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:14:53.559670 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556186 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:14:53.559670 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556188 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:14:53.559670 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556191 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:14:53.559670 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556193 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:14:53.559670 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556196 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:14:53.559670 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556199 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:14:53.559670 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556201 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:14:53.560133 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556204 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:14:53.560133 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556206 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:14:53.560133 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556209 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:14:53.560133 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556211 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:14:53.560133 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556214 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:14:53.560133 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556217 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:14:53.560133 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556220 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:14:53.560133 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556223 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:14:53.560133 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556225 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:14:53.560133 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556228 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:14:53.560133 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556232 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:14:53.560133 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556237 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:14:53.560133 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556240 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:14:53.560133 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.556243 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:14:53.560133 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.557989 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:14:53.560133 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558121 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:14:53.560133 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558126 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:14:53.560133 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558130 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:14:53.560133 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558134 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:14:53.560133 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558137 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:14:53.560615 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558142 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:14:53.560615 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558145 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:14:53.560615 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558148 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:14:53.560615 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558151 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:14:53.560615 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558154 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:14:53.560615 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558157 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:14:53.560615 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558161 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:14:53.560615 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558164 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:14:53.560615 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558167 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:14:53.560615 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558171 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:14:53.560615 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558174 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:14:53.560615 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558177 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:14:53.560615 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558180 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:14:53.560615 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558183 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:14:53.560615 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558186 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:14:53.560615 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558188 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:14:53.560615 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558191 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:14:53.560615 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558194 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:14:53.560615 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558196 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:14:53.560615 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558199 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:14:53.561109 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558202 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:14:53.561109 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558205 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:14:53.561109 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558207 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:14:53.561109 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558210 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:14:53.561109 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558213 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:14:53.561109 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558216 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:14:53.561109 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558218 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:14:53.561109 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558223 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:14:53.561109 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558226 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:14:53.561109 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558229 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:14:53.561109 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558232 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:14:53.561109 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558235 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:14:53.561109 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558238 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:14:53.561109 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558241 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:14:53.561109 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558244 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:14:53.561109 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558247 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:14:53.561109 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558249 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:14:53.561109 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558252 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:14:53.561109 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558259 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:14:53.561555 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558263 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:14:53.561555 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558267 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:14:53.561555 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558270 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:14:53.561555 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558273 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:14:53.561555 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558277 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:14:53.561555 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558281 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:14:53.561555 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558284 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:14:53.561555 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558288 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:14:53.561555 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558292 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:14:53.561555 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558295 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:14:53.561555 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558299 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:14:53.561555 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558302 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:14:53.561555 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558305 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:14:53.561555 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558307 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:14:53.561555 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558310 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:14:53.561555 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558312 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:14:53.561555 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558316 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:14:53.561555 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558318 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:14:53.561555 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558321 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:14:53.562002 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558324 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:14:53.562002 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558328 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:14:53.562002 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558331 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:14:53.562002 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558334 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:14:53.562002 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558336 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:14:53.562002 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558339 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:14:53.562002 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558342 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:14:53.562002 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558344 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:14:53.562002 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558347 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:14:53.562002 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558350 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:14:53.562002 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558352 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:14:53.562002 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558355 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:14:53.562002 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558358 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:14:53.562002 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558360 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:14:53.562002 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558363 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:14:53.562002 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558365 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:14:53.562002 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558368 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:14:53.562002 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558371 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:14:53.562002 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558373 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:14:53.562002 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558376 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:14:53.562524 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558378 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:14:53.562524 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558381 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:14:53.562524 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558383 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:14:53.562524 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558386 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:14:53.562524 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558389 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:14:53.562524 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558391 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:14:53.562524 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558394 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:14:53.562524 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558397 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:14:53.562524 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558399 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:14:53.562524 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558402 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:14:53.562524 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558405 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:14:53.562524 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558407 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:14:53.562524 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558410 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:14:53.562524 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558413 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:14:53.562524 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558416 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:14:53.562524 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558419 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:14:53.562524 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558422 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:14:53.562524 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558424 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:14:53.562524 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558427 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:14:53.562524 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558430 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558432 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.558436 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559580 2569 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559594 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559601 2569 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559607 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559611 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559615 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559620 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559624 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559628 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559631 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559636 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559640 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559643 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559646 2569 flags.go:64] FLAG: --cgroup-root="" Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559649 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559653 2569 flags.go:64] FLAG: --client-ca-file="" Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559656 2569 flags.go:64] FLAG: --cloud-provider="external" Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559660 2569 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559665 2569 flags.go:64] FLAG: --cluster-domain="" Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559668 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 03:14:53.563016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559672 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559689 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559694 2569 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559701 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559705 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559709 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559712 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559715 2569 flags.go:64] FLAG: --contention-profiling="false" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559719 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559722 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559725 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559728 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559733 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559736 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559739 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559742 2569 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559746 2569 flags.go:64] FLAG: --enable-server="true" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559749 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559754 2569 flags.go:64] FLAG: --event-burst="100" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559758 2569 flags.go:64] FLAG: --event-qps="50" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559761 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559764 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559767 2569 flags.go:64] FLAG: --eviction-hard="" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559773 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 03:14:53.563606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559776 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559780 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559783 2569 flags.go:64] FLAG: --eviction-soft="" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559787 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559791 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559794 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559797 2569 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559800 2569 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559803 2569 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559806 2569 flags.go:64] FLAG: --feature-gates="" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559811 2569 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559815 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559818 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559821 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559824 2569 flags.go:64] FLAG: --healthz-port="10248" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559828 2569 flags.go:64] FLAG: --help="false" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559831 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559834 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559837 2569 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559840 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559844 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559847 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559850 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 03:14:53.564237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559853 2569 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559856 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559859 2569 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559863 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559866 2569 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559869 2569 flags.go:64] FLAG: --kube-reserved="" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559872 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559875 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559879 2569 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559882 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559885 2569 flags.go:64] FLAG: --lock-file="" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559888 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559891 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559895 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559901 2569 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559904 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559907 2569 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559910 2569 flags.go:64] FLAG: --logging-format="text" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559913 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559920 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559924 2569 flags.go:64] FLAG: --manifest-url="" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559927 2569 flags.go:64] FLAG: --manifest-url-header="" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559932 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559935 2569 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559939 2569 flags.go:64] FLAG: --max-pods="110" Oct 31 03:14:53.564803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559942 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559946 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559949 2569 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559952 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559955 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559958 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559961 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559970 2569 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559973 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559976 2569 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559979 2569 flags.go:64] FLAG: --pod-cidr="" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559982 2569 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559985 2569 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559989 2569 flags.go:64] FLAG: --pods-per-core="0" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559992 2569 flags.go:64] FLAG: --port="10250" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559995 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.559998 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0148485198cd11abf" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560002 2569 flags.go:64] FLAG: --qos-reserved="" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560006 2569 flags.go:64] FLAG: --read-only-port="10255" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560009 2569 flags.go:64] FLAG: --register-node="true" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560012 2569 flags.go:64] FLAG: --register-with-taints="" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560017 2569 flags.go:64] FLAG: --registry-burst="10" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560020 2569 flags.go:64] FLAG: --registry-qps="5" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560023 2569 flags.go:64] FLAG: --reserved-cpus="" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560026 2569 flags.go:64] FLAG: --reserved-memory="" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560030 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 03:14:53.565399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560035 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560039 2569 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560042 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560045 2569 flags.go:64] FLAG: --runonce="false" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560048 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560051 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560054 2569 flags.go:64] FLAG: --seccomp-default="false" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560057 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560060 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560064 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560067 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560071 2569 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560074 2569 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560077 2569 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560080 2569 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560083 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560087 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560090 2569 flags.go:64] FLAG: --system-cgroups="" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560093 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560099 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560102 2569 flags.go:64] FLAG: --tls-cert-file="" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560106 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560136 2569 flags.go:64] FLAG: --tls-min-version="" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560140 2569 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560144 2569 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 03:14:53.566073 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560147 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560151 2569 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560154 2569 flags.go:64] FLAG: --v="2" Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560159 2569 flags.go:64] FLAG: --version="false" Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560164 2569 flags.go:64] FLAG: --vmodule="" Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560169 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560172 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560326 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560331 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560334 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560337 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560340 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560343 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560346 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560348 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560351 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560354 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560357 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560359 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560362 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560365 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:14:53.566705 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560368 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:14:53.567200 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560371 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:14:53.567200 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560374 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:14:53.567200 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560377 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:14:53.567200 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560380 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:14:53.567200 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560382 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:14:53.567200 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560385 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:14:53.567200 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560388 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:14:53.567200 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560391 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:14:53.567200 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560393 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:14:53.567200 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560396 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:14:53.567200 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560402 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:14:53.567200 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560405 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:14:53.567200 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560408 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:14:53.567200 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560410 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:14:53.567200 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560413 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:14:53.567200 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560416 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:14:53.567200 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560419 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:14:53.567200 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560423 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:14:53.567200 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560425 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:14:53.567658 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560428 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:14:53.567658 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560431 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:14:53.567658 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560434 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:14:53.567658 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560436 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:14:53.567658 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560439 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:14:53.567658 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560442 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:14:53.567658 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560444 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:14:53.567658 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560447 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:14:53.567658 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560450 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:14:53.567658 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560453 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:14:53.567658 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560456 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:14:53.567658 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560458 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:14:53.567658 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560461 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:14:53.567658 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560464 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:14:53.567658 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560466 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:14:53.567658 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560469 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:14:53.567658 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560471 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:14:53.567658 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560474 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:14:53.567658 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560477 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:14:53.568207 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560480 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:14:53.568207 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560482 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:14:53.568207 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560485 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:14:53.568207 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560488 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:14:53.568207 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560492 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:14:53.568207 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560495 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:14:53.568207 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560498 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:14:53.568207 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560501 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:14:53.568207 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560503 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:14:53.568207 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560506 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:14:53.568207 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560509 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:14:53.568207 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560513 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:14:53.568207 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560516 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:14:53.568207 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560519 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:14:53.568207 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560521 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:14:53.568207 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560524 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:14:53.568207 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560527 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:14:53.568207 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560529 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:14:53.568207 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560532 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:14:53.568207 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560535 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:14:53.568699 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560537 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:14:53.568699 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560540 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:14:53.568699 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560544 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:14:53.568699 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560548 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:14:53.568699 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560551 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:14:53.568699 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560554 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:14:53.568699 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560557 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:14:53.568699 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560560 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:14:53.568699 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560563 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:14:53.568699 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560565 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:14:53.568699 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560568 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:14:53.568699 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560571 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:14:53.568699 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560573 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:14:53.568699 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560576 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:14:53.568699 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560580 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:14:53.568699 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560583 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:14:53.568699 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560588 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:14:53.568699 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560591 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:14:53.568699 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560594 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:14:53.569165 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560597 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:14:53.569165 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560600 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:14:53.569165 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560603 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:14:53.569165 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560605 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:14:53.569165 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560609 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:14:53.569165 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560612 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:14:53.569165 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560615 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:14:53.569165 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560618 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:14:53.569165 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560621 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:14:53.569165 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560624 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:14:53.569165 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560627 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:14:53.569165 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560630 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:14:53.569165 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560632 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:14:53.569165 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560635 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:14:53.569165 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.560663 2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:14:53.569165 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560719 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:14:53.569534 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560725 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:14:53.569534 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560728 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:14:53.569534 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560731 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:14:53.569534 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560734 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:14:53.569534 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560737 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:14:53.569534 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560740 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:14:53.569534 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560743 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:14:53.569534 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560745 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:14:53.569534 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560748 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:14:53.569534 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560751 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:14:53.569534 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560755 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:14:53.569534 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560758 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:14:53.569534 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560761 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:14:53.569534 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560767 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:14:53.569534 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560770 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:14:53.569534 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560773 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:14:53.569534 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560776 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:14:53.569534 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560779 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:14:53.569534 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560782 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:14:53.569992 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560785 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:14:53.569992 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560788 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:14:53.569992 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560791 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:14:53.569992 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560794 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:14:53.569992 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560797 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:14:53.569992 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560799 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:14:53.569992 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560802 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:14:53.569992 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560805 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:14:53.569992 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560807 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:14:53.569992 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560810 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:14:53.569992 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560813 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:14:53.569992 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560815 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:14:53.569992 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560818 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:14:53.569992 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560821 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:14:53.569992 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560823 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:14:53.569992 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560826 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:14:53.569992 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560829 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:14:53.569992 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560832 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:14:53.569992 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560835 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:14:53.569992 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560838 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:14:53.570460 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560841 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:14:53.570460 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560843 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:14:53.570460 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560846 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:14:53.570460 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560849 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:14:53.570460 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560852 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:14:53.570460 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560854 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:14:53.570460 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560858 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:14:53.570460 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560861 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:14:53.570460 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560864 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:14:53.570460 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560867 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:14:53.570460 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560870 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:14:53.570460 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560880 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:14:53.570460 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560884 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:14:53.570460 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560887 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:14:53.570460 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560891 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:14:53.570460 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560894 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:14:53.570460 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560897 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:14:53.570460 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560900 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:14:53.570460 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560903 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:14:53.570999 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560906 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:14:53.570999 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560908 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:14:53.570999 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560911 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:14:53.570999 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560914 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:14:53.570999 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560916 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:14:53.570999 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560919 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:14:53.570999 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560922 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:14:53.570999 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560924 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:14:53.570999 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560927 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:14:53.570999 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560930 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:14:53.570999 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560932 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:14:53.570999 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560935 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:14:53.570999 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560938 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:14:53.570999 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560940 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:14:53.570999 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560943 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:14:53.570999 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560946 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:14:53.570999 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560949 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:14:53.570999 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560952 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:14:53.570999 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560955 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:14:53.571467 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560958 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:14:53.571467 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560961 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:14:53.571467 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560964 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:14:53.571467 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560966 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:14:53.571467 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560969 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:14:53.571467 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560973 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:14:53.571467 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560976 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:14:53.571467 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560978 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:14:53.571467 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560982 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:14:53.571467 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560985 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:14:53.571467 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560987 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:14:53.571467 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560991 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:14:53.571467 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560993 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:14:53.571467 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560996 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:14:53.571467 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.560999 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:14:53.571467 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.561001 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:14:53.571467 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.561004 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:14:53.571467 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.561007 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:14:53.571467 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.561010 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:14:53.571467 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.561013 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:14:53.571958 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.561015 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:14:53.571958 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.561018 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:14:53.571958 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.561021 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:14:53.571958 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.561024 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:14:53.571958 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.561026 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:14:53.571958 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.561029 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:14:53.571958 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.561032 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:14:53.571958 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.561036 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:14:53.571958 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.568300 2569 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 03:14:53.571958 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.568317 2569 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 03:14:53.571958 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.569097 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 03:14:53.571958 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.569104 2569 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 03:14:53.571958 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569180 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:14:53.571958 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569187 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:14:53.571958 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569190 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:14:53.571958 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569193 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:14:53.571958 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569196 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:14:53.571958 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569199 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:14:53.571958 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569201 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:14:53.572409 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569205 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:14:53.572409 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569207 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:14:53.572409 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569210 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:14:53.572409 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569213 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:14:53.572409 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569216 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:14:53.572409 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569219 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:14:53.572409 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569222 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:14:53.572409 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569224 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:14:53.572409 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569227 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:14:53.572409 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569230 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:14:53.572409 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569233 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:14:53.572409 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569236 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:14:53.572409 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569239 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:14:53.572409 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569241 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:14:53.572409 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569245 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:14:53.572409 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569247 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:14:53.572409 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569250 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:14:53.572409 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569253 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:14:53.572409 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569255 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:14:53.572409 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569258 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:14:53.572901 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569261 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:14:53.572901 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569264 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:14:53.572901 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569269 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:14:53.572901 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569273 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:14:53.572901 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569277 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:14:53.572901 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569280 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:14:53.572901 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569283 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:14:53.572901 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569285 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:14:53.572901 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569288 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:14:53.572901 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569291 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:14:53.572901 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569294 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:14:53.572901 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569296 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:14:53.572901 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569299 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:14:53.572901 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569302 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:14:53.572901 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569305 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:14:53.572901 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569308 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:14:53.572901 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569311 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:14:53.572901 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569313 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:14:53.572901 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569316 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:14:53.572901 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569319 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:14:53.573361 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569323 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:14:53.573361 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569326 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:14:53.573361 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569328 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:14:53.573361 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569331 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:14:53.573361 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569334 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:14:53.573361 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569337 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:14:53.573361 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569340 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:14:53.573361 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569343 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:14:53.573361 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569346 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:14:53.573361 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569349 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:14:53.573361 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569352 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:14:53.573361 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569355 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:14:53.573361 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569357 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:14:53.573361 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569360 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:14:53.573361 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569362 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:14:53.573361 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569366 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:14:53.573361 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569369 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:14:53.573361 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569371 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:14:53.573361 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569374 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:14:53.573361 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569377 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:14:53.573837 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569379 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:14:53.573837 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569382 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:14:53.573837 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569385 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:14:53.573837 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569387 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:14:53.573837 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569390 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:14:53.573837 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569393 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:14:53.573837 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569395 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:14:53.573837 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569398 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:14:53.573837 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569401 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:14:53.573837 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569404 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:14:53.573837 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569406 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:14:53.573837 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569409 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:14:53.573837 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569412 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:14:53.573837 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569415 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:14:53.573837 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569418 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:14:53.573837 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569420 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:14:53.573837 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569425 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:14:53.573837 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569428 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:14:53.573837 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569432 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:14:53.574281 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569434 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:14:53.574281 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569437 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:14:53.574281 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569440 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:14:53.574281 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569443 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:14:53.574281 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569446 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:14:53.574281 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569448 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:14:53.574281 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569451 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:14:53.574281 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569454 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:14:53.574281 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569456 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:14:53.574281 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569460 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:14:53.574281 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569462 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:14:53.574281 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569465 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:14:53.574281 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569468 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:14:53.574281 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569470 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:14:53.574281 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569473 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:14:53.574281 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569475 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:14:53.574281 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569478 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:14:53.574281 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569480 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:14:53.574281 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569483 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:14:53.574809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569486 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:14:53.574809 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.569514 2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:14:53.574809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569660 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:14:53.574809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569665 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:14:53.574809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569669 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:14:53.574809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569671 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:14:53.574809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569685 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:14:53.574809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569688 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:14:53.574809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569691 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:14:53.574809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569694 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:14:53.574809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569697 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:14:53.574809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569700 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:14:53.574809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569703 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:14:53.574809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569706 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:14:53.574809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569709 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:14:53.574809 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569712 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:14:53.575211 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569715 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:14:53.575211 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569717 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:14:53.575211 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569720 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:14:53.575211 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569723 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:14:53.575211 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569726 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:14:53.575211 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569728 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:14:53.575211 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569731 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:14:53.575211 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569735 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:14:53.575211 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569739 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:14:53.575211 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569742 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:14:53.575211 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569745 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:14:53.575211 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569748 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:14:53.575211 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569751 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:14:53.575211 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569753 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:14:53.575211 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569756 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:14:53.575211 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569759 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:14:53.575211 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569762 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:14:53.575211 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569764 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:14:53.575211 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569767 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:14:53.575211 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569770 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:14:53.575697 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569773 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:14:53.575697 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569776 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:14:53.575697 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569779 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:14:53.575697 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569781 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:14:53.575697 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569784 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:14:53.575697 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569787 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:14:53.575697 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569789 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:14:53.575697 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569792 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:14:53.575697 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569795 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:14:53.575697 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569798 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:14:53.575697 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569800 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:14:53.575697 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569803 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:14:53.575697 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569806 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:14:53.575697 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569809 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:14:53.575697 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569811 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:14:53.575697 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569814 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:14:53.575697 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569817 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:14:53.575697 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569821 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:14:53.575697 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569824 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:14:53.576145 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569827 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:14:53.576145 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569831 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:14:53.576145 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569834 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:14:53.576145 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569837 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:14:53.576145 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569839 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:14:53.576145 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569842 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:14:53.576145 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569845 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:14:53.576145 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569847 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:14:53.576145 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569850 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:14:53.576145 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569853 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:14:53.576145 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569855 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:14:53.576145 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569858 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:14:53.576145 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569861 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:14:53.576145 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569864 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:14:53.576145 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569866 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:14:53.576145 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569869 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:14:53.576145 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569872 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:14:53.576145 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569875 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:14:53.576145 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569877 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:14:53.576145 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569880 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:14:53.576614 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569883 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:14:53.576614 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569886 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:14:53.576614 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569888 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:14:53.576614 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569891 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:14:53.576614 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569894 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:14:53.576614 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569896 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:14:53.576614 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569899 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:14:53.576614 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569902 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:14:53.576614 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569904 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:14:53.576614 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569907 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:14:53.576614 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569910 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:14:53.576614 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569912 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:14:53.576614 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569915 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:14:53.576614 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569918 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:14:53.576614 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569921 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:14:53.576614 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569923 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:14:53.576614 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569926 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:14:53.576614 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569929 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:14:53.576614 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569931 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:14:53.577099 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569934 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:14:53.577099 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569937 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:14:53.577099 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569940 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:14:53.577099 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569943 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:14:53.577099 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569946 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:14:53.577099 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569948 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:14:53.577099 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569951 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:14:53.577099 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569954 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:14:53.577099 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569956 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:14:53.577099 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569959 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:14:53.577099 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569962 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:14:53.577099 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569964 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:14:53.577099 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569967 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:14:53.577099 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.569970 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:14:53.577099 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.569994 2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:14:53.577446 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570035 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:14:53.577446 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570038 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:14:53.577446 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570043 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:14:53.577446 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570047 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:14:53.577446 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570050 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:14:53.577446 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570053 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:14:53.577446 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570056 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:14:53.577446 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570059 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:14:53.577446 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570062 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:14:53.577446 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570064 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:14:53.577446 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570067 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:14:53.577446 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570070 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:14:53.577446 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570073 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:14:53.577446 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570076 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:14:53.577446 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570078 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:14:53.577446 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570081 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:14:53.577446 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570084 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:14:53.577446 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570087 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:14:53.577446 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570092 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:14:53.577898 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570094 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:14:53.577898 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570097 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:14:53.577898 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570100 2569 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:14:53.577898 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570102 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:14:53.577898 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570105 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:14:53.577898 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570108 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:14:53.577898 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570111 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:14:53.577898 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570113 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:14:53.577898 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570116 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:14:53.577898 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570119 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:14:53.577898 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570121 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:14:53.577898 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570124 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:14:53.577898 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570127 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:14:53.577898 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570130 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:14:53.577898 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570133 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:14:53.577898 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570135 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:14:53.577898 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570138 2569 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:14:53.577898 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570141 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:14:53.577898 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570143 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:14:53.578387 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570146 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:14:53.578387 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570148 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:14:53.578387 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570151 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:14:53.578387 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570154 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:14:53.578387 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570156 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:14:53.578387 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570159 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:14:53.578387 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570162 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:14:53.578387 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570164 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:14:53.578387 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570168 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:14:53.578387 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570170 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:14:53.578387 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570173 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:14:53.578387 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570176 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:14:53.578387 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570180 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:14:53.578387 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570183 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:14:53.578387 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570186 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:14:53.578387 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570188 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:14:53.578387 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570191 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:14:53.578387 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570194 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:14:53.578387 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570197 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:14:53.578841 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570199 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:14:53.578841 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570202 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:14:53.578841 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570205 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:14:53.578841 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570207 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:14:53.578841 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570210 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:14:53.578841 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570213 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:14:53.578841 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570216 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:14:53.578841 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570219 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:14:53.578841 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570221 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:14:53.578841 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570224 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:14:53.578841 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570227 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:14:53.578841 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570230 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:14:53.578841 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570232 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:14:53.578841 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570235 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:14:53.578841 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570237 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:14:53.578841 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570240 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:14:53.578841 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570243 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:14:53.578841 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570245 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:14:53.578841 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570248 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:14:53.578841 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570251 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:14:53.579357 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570253 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:14:53.579357 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570258 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:14:53.579357 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570262 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:14:53.579357 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570265 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:14:53.579357 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570269 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:14:53.579357 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570273 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:14:53.579357 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570276 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:14:53.579357 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570279 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:14:53.579357 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570282 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:14:53.579357 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570284 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:14:53.579357 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570287 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:14:53.579357 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570290 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:14:53.579357 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570293 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:14:53.579357 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570295 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:14:53.579357 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570298 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:14:53.579357 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570301 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:14:53.579357 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570303 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:14:53.579357 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570306 2569 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:14:53.579357 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570309 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:14:53.579357 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570312 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:14:53.579914 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570315 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:14:53.579914 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570317 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:14:53.579914 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570320 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:14:53.579914 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570323 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:14:53.579914 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570325 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:14:53.579914 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570328 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:14:53.579914 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570332 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:14:53.579914 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570335 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:14:53.579914 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:53.570337 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:14:53.579914 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.570401 2569 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 03:14:53.579914 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.574647 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 03:14:53.579914 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.575800 2569 server.go:1014] "Starting client certificate rotation" Oct 31 03:14:53.579914 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.575896 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 03:14:53.579914 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.575925 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 03:14:53.579914 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.578193 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:14:53.579914 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.579399 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:14:53.588359 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.588334 2569 log.go:25] "Validated CRI v1 runtime API" Oct 31 03:14:53.593925 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.593906 2569 log.go:25] "Validated CRI v1 image API" Oct 31 03:14:53.594817 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.594784 2569 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 03:14:53.598376 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.598354 2569 fs.go:135] Filesystem UUIDs: map[2cd556e6-930f-4dae-9d56-d5647d119e3f:/dev/nvme0n1p3 65e4edfa-7282-429b-8d3b-c91b929ed9d6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 31 03:14:53.598452 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.598375 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 03:14:53.602712 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.602573 2569 manager.go:217] Machine: {Timestamp:2026-10-31 03:14:53.602096275 +0000 UTC m=+0.334529338 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106569 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22067afb54a9236ebcbc456fcdeb33 SystemUUID:ec22067a-fb54-a923-6ebc-bc456fcdeb33 BootID:4a3470f5-09fb-46a8-a9b3-843ec9a4c60b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cc:5f:23:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cc:5f:23:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:c1:c5:09:a2:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 03:14:53.602712 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.602701 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 03:14:53.602869 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.602784 2569 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 03:14:53.606458 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.606425 2569 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 03:14:53.606599 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.606460 2569 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-76.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 03:14:53.606644 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.606602 2569 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 03:14:53.606644 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.606610 2569 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 03:14:53.606644 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.606629 2569 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 03:14:53.606644 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.606642 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 03:14:53.607941 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.607930 2569 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 03:14:53.608389 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.608373 2569 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 03:14:53.612592 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.612579 2569 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 03:14:53.612771 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.612761 2569 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 03:14:53.612819 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.612796 2569 kubelet.go:493] "Attempting to sync node with API server" Oct 31 03:14:53.612819 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.612807 2569 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 03:14:53.612819 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.612817 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 03:14:53.612933 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.612827 2569 kubelet.go:405] "Adding apiserver pod source" Oct 31 03:14:53.612933 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.612837 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 03:14:53.613264 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.613248 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 03:14:53.615885 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.615865 2569 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 03:14:53.622670 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.622652 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 03:14:53.623535 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.623519 2569 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 03:14:53.623612 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.623561 2569 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 03:14:53.623801 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.623785 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 03:14:53.623862 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.623807 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 03:14:53.623862 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.623817 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 03:14:53.623862 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.623826 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 03:14:53.623862 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.623835 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 03:14:53.623862 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.623845 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 03:14:53.624532 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.624518 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 03:14:53.624599 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.624534 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 03:14:53.624599 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.624547 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 03:14:53.624599 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.624557 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 03:14:53.624599 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.624571 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 03:14:53.624599 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.624586 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 03:14:53.625491 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.625475 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 03:14:53.625574 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.625495 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 03:14:53.629603 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.629586 2569 server.go:1263] "Started kubelet" Oct 31 03:14:53.629744 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.629687 2569 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 03:14:53.629744 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.629690 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 03:14:53.629851 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.629761 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 03:14:53.630471 ip-10-0-142-76 systemd[1]: Started Kubernetes Kubelet. Oct 31 03:14:53.631336 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.631322 2569 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 03:14:53.636043 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.635997 2569 server.go:317] "Adding debug handlers to kubelet server" Oct 31 03:14:53.636263 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.636239 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7g4kb" Oct 31 03:14:53.637196 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:53.637171 2569 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 03:14:53.638074 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.638056 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-76.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 03:14:53.638202 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:53.638082 2569 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 03:14:53.639143 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:53.638011 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-76.us-east-2.compute.internal.18e37ecc8d1b2809 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-76.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-76.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 03:14:53.629384713 +0000 UTC m=+0.361817763,LastTimestamp:2026-10-31 03:14:53.629384713 +0000 UTC m=+0.361817763,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-76.us-east-2.compute.internal,}" Oct 31 03:14:53.641078 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.641060 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 03:14:53.641716 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.641699 2569 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 03:14:53.641868 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.641849 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7g4kb" Oct 31 03:14:53.642142 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:53.642124 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-76.us-east-2.compute.internal\" not found" Oct 31 03:14:53.642483 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.642430 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 03:14:53.642483 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.642439 2569 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 03:14:53.642483 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.642461 2569 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 03:14:53.642633 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.642544 2569 reconstruct.go:97] "Volume reconstruction finished" Oct 31 03:14:53.642633 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.642555 2569 reconciler.go:29] "Reconciler: start to sync state" Oct 31 03:14:53.642813 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.642790 2569 factory.go:55] Registering systemd factory Oct 31 03:14:53.642813 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.642807 2569 factory.go:223] Registration of the systemd container factory successfully Oct 31 03:14:53.643080 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.643062 2569 factory.go:153] Registering CRI-O factory Oct 31 03:14:53.643080 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.643079 2569 factory.go:223] Registration of the crio container factory successfully Oct 31 03:14:53.643217 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.643127 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 03:14:53.643217 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.643141 2569 factory.go:103] Registering Raw factory Oct 31 03:14:53.643217 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.643153 2569 manager.go:1199] Started watching for new ooms in manager Oct 31 03:14:53.644252 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.644230 2569 manager.go:319] Starting recovery of all containers Oct 31 03:14:53.650318 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:53.650271 2569 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-76.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 03:14:53.650416 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:53.650323 2569 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-76.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 03:14:53.655872 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.655570 2569 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:14:53.655934 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.655867 2569 manager.go:324] Recovery completed Oct 31 03:14:53.660410 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.660397 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:14:53.663091 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.663071 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:14:53.663159 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.663106 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:14:53.663159 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.663119 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:14:53.663714 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.663699 2569 cpu_manager.go:226] "Starting" policy="none" Oct 31 03:14:53.663714 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.663712 2569 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 03:14:53.663714 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.663732 2569 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 03:14:53.663861 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.663707 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:14:53.664524 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.664510 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:14:53.664575 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.664541 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:14:53.664575 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.664554 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:14:53.665745 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.665732 2569 policy_none.go:50] "Start" Oct 31 03:14:53.665784 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.665751 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 03:14:53.665784 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.665765 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 03:14:53.667782 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.667766 2569 policy_none.go:44] "Start" Oct 31 03:14:53.725600 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.725560 2569 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 03:14:53.728200 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.728182 2569 manager.go:341] "Starting Device Plugin manager" Oct 31 03:14:53.728298 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:53.728232 2569 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 03:14:53.728298 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.728244 2569 server.go:87] "Starting device plugin registration server" Oct 31 03:14:53.728533 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.728518 2569 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 03:14:53.728588 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.728534 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 03:14:53.728935 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.728780 2569 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 03:14:53.729153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.729004 2569 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 03:14:53.729153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.729012 2569 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 03:14:53.729512 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:53.729493 2569 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 03:14:53.729704 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:53.729548 2569 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-76.us-east-2.compute.internal\" not found" Oct 31 03:14:53.778507 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.778472 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 03:14:53.780559 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.779729 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 03:14:53.780559 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.779747 2569 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 03:14:53.780559 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.779772 2569 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 03:14:53.780559 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:53.779808 2569 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 03:14:53.783142 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.783122 2569 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:14:53.829666 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.829581 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:14:53.830643 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.830626 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:14:53.830764 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.830662 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:14:53.830764 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.830692 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:14:53.830764 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.830727 2569 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:14:53.838504 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.838485 2569 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:14:53.838587 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:53.838513 2569 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-76.us-east-2.compute.internal\": node \"ip-10-0-142-76.us-east-2.compute.internal\" not found" Oct 31 03:14:53.859457 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:53.859426 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-76.us-east-2.compute.internal\" not found" Oct 31 03:14:53.880794 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.880757 2569 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal"] Oct 31 03:14:53.880948 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.880827 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:14:53.881893 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.881871 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:14:53.881997 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.881901 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:14:53.881997 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.881912 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:14:53.884127 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.884114 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:14:53.884271 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.884254 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:14:53.884305 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.884284 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:14:53.884921 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.884901 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:14:53.885017 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.884935 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:14:53.885017 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.884946 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:14:53.885017 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.884909 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:14:53.885017 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.885005 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:14:53.885017 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.885016 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:14:53.887112 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.887094 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:14:53.887178 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.887125 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:14:53.887931 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.887916 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:14:53.888020 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.887947 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:14:53.888020 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:53.887961 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:14:53.920737 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:53.920710 2569 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-76.us-east-2.compute.internal\" not found" node="ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:14:53.927036 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:53.927014 2569 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-76.us-east-2.compute.internal\" not found" node="ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:14:53.960313 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:53.960280 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-76.us-east-2.compute.internal\" not found" Oct 31 03:14:54.043934 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:54.043898 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9dcc8bbdd4365b42723f25cd49f22a1c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal\" (UID: \"9dcc8bbdd4365b42723f25cd49f22a1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:14:54.044096 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:54.043935 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dcc8bbdd4365b42723f25cd49f22a1c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal\" (UID: \"9dcc8bbdd4365b42723f25cd49f22a1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:14:54.044096 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:54.043966 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc792e750c2606b177de944a501bf339-config\") pod \"kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal\" (UID: \"fc792e750c2606b177de944a501bf339\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:14:54.061023 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:54.060996 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-76.us-east-2.compute.internal\" not found" Oct 31 03:14:54.144727 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:54.144639 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc792e750c2606b177de944a501bf339-config\") pod \"kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal\" (UID: \"fc792e750c2606b177de944a501bf339\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:14:54.144727 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:54.144690 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9dcc8bbdd4365b42723f25cd49f22a1c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal\" (UID: \"9dcc8bbdd4365b42723f25cd49f22a1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:14:54.144727 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:54.144713 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dcc8bbdd4365b42723f25cd49f22a1c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal\" (UID: \"9dcc8bbdd4365b42723f25cd49f22a1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:14:54.144926 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:54.144761 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dcc8bbdd4365b42723f25cd49f22a1c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal\" (UID: \"9dcc8bbdd4365b42723f25cd49f22a1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:14:54.144926 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:54.144763 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc792e750c2606b177de944a501bf339-config\") pod \"kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal\" (UID: \"fc792e750c2606b177de944a501bf339\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:14:54.144926 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:54.144762 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9dcc8bbdd4365b42723f25cd49f22a1c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal\" (UID: \"9dcc8bbdd4365b42723f25cd49f22a1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:14:54.161838 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:54.161794 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-76.us-east-2.compute.internal\" not found" Oct 31 03:14:54.226315 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:54.226276 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:14:54.229710 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:54.229666 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:14:54.263159 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:54.263121 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-76.us-east-2.compute.internal\" not found" Oct 31 03:14:54.364075 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:54.364019 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-76.us-east-2.compute.internal\" not found" Oct 31 03:14:54.464438 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:54.464356 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-76.us-east-2.compute.internal\" not found" Oct 31 03:14:54.564659 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:54.564628 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-76.us-east-2.compute.internal\" not found" Oct 31 03:14:54.575621 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:54.575576 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 03:14:54.575838 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:54.575810 2569 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 03:14:54.575885 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:54.575868 2569 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 03:14:54.651067 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:54.651009 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:09:54 +0000 UTC" deadline="2028-03-31 02:42:09.965316876 +0000 UTC" Oct 31 03:14:54.651067 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:54.651057 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12407h27m15.314262467s" Oct 31 03:14:54.664768 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:54.664737 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-76.us-east-2.compute.internal\" not found" Oct 31 03:14:54.765438 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:54.765338 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-76.us-east-2.compute.internal\" not found" Oct 31 03:14:54.866539 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:54.866502 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-76.us-east-2.compute.internal\" not found" Oct 31 03:14:54.936147 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:54.936116 2569 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:14:54.967032 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:54.966993 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-76.us-east-2.compute.internal\" not found" Oct 31 03:14:55.003837 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:55.003664 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dcc8bbdd4365b42723f25cd49f22a1c.slice/crio-ab39d17504c1ed393c39b9ff8d73ce1229730f78105dc51229457d973aaf7a67 WatchSource:0}: Error finding container ab39d17504c1ed393c39b9ff8d73ce1229730f78105dc51229457d973aaf7a67: Status 404 returned error can't find the container with id ab39d17504c1ed393c39b9ff8d73ce1229730f78105dc51229457d973aaf7a67 Oct 31 03:14:55.012141 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.012120 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:14:55.015356 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:55.015332 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc792e750c2606b177de944a501bf339.slice/crio-9464aa9bf3b2aa43daa2a9323f4fe8a5b4c275c88cedd1e89755a98a3df10be7 WatchSource:0}: Error finding container 9464aa9bf3b2aa43daa2a9323f4fe8a5b4c275c88cedd1e89755a98a3df10be7: Status 404 returned error can't find the container with id 9464aa9bf3b2aa43daa2a9323f4fe8a5b4c275c88cedd1e89755a98a3df10be7 Oct 31 03:14:55.068018 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:55.067975 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-76.us-east-2.compute.internal\" not found" Oct 31 03:14:55.125685 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.125656 2569 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:14:55.143023 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.142981 2569 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:14:55.153589 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.153565 2569 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:14:55.154456 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.154441 2569 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:14:55.166653 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.166627 2569 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:14:55.616435 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.616400 2569 apiserver.go:52] "Watching apiserver" Oct 31 03:14:55.624200 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.624170 2569 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 03:14:55.624504 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.624479 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-5lds2","openshift-dns/node-resolver-z74z2","openshift-image-registry/node-ca-g2mdn","openshift-multus/network-metrics-daemon-lkp2n","kube-system/global-pull-secret-syncer-gkf5m","openshift-multus/multus-additional-cni-plugins-zcfc2","kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb","openshift-cluster-node-tuning-operator/tuned-4hwhg","openshift-multus/multus-h9wtm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-4tzpt","openshift-network-operator/iptables-alerter-frjws","openshift-ovn-kubernetes/ovnkube-node-x6rwl"] Oct 31 03:14:55.627244 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.627211 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.631265 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.631082 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 03:14:55.631265 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.631090 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 03:14:55.631265 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.631094 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 03:14:55.631265 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.631143 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 03:14:55.632848 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.631738 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-b55zh\"" Oct 31 03:14:55.632984 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.632958 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-frjws" Oct 31 03:14:55.633978 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.633957 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g2mdn" Oct 31 03:14:55.636313 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.636289 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:14:55.636470 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:55.636445 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:14:55.636564 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.636506 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:14:55.638483 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.638460 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:14:55.638589 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:55.638535 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:14:55.640995 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.640852 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.641732 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.641712 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 03:14:55.644555 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.643249 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.646531 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.646505 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.649010 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.648992 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z74z2" Oct 31 03:14:55.651789 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.651765 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5lds2" Oct 31 03:14:55.653755 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.653729 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkr86\" (UniqueName: \"kubernetes.io/projected/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-kube-api-access-dkr86\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.653864 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.653775 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-multus-cni-dir\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.653864 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.653807 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-multus-socket-dir-parent\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.653864 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.653834 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d877fe5-7b94-4645-bff5-6df0796bb637-dbus\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:14:55.654009 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.653861 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-modprobe-d\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.654009 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.653887 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-tuned\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.654009 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.653959 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-tmp\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.654009 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654000 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-cnibin\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.654218 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654096 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/904f42d7-69e3-462f-9055-b6b840800f13-multus-daemon-config\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.654218 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654129 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-lib-modules\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.654218 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654155 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z82fk\" (UniqueName: \"kubernetes.io/projected/55f1a18a-59ef-4146-a0e8-3fc632ab20f5-kube-api-access-z82fk\") pod \"node-ca-g2mdn\" (UID: \"55f1a18a-59ef-4146-a0e8-3fc632ab20f5\") " pod="openshift-image-registry/node-ca-g2mdn" Oct 31 03:14:55.654218 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654186 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qsw4\" (UniqueName: \"kubernetes.io/projected/d7a31b67-a1b9-4b6a-a112-69cd5c833019-kube-api-access-2qsw4\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.654218 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654212 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-run-k8s-cni-cncf-io\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.654471 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654235 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-var-lib-kubelet\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.654471 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654259 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-etc-kubernetes\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.654471 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654294 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/571b679b-fad2-423d-96e5-8d5550bb536d-host-slash\") pod \"iptables-alerter-frjws\" (UID: \"571b679b-fad2-423d-96e5-8d5550bb536d\") " pod="openshift-network-operator/iptables-alerter-frjws" Oct 31 03:14:55.654471 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654319 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:14:55.654471 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654343 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-sysctl-conf\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.654471 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654366 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w65q\" (UniqueName: \"kubernetes.io/projected/d680db8a-15e4-4817-bff1-a420f9f54681-kube-api-access-6w65q\") pod \"network-metrics-daemon-lkp2n\" (UID: \"d680db8a-15e4-4817-bff1-a420f9f54681\") " pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:14:55.654471 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654391 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-run-netns\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.654471 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654416 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-var-lib-cni-bin\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.654471 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654441 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-multus-conf-dir\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.654471 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654466 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-run-multus-certs\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.655006 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654491 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-systemd\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.655006 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654514 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-var-lib-kubelet\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.655006 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654541 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/55f1a18a-59ef-4146-a0e8-3fc632ab20f5-serviceca\") pod \"node-ca-g2mdn\" (UID: \"55f1a18a-59ef-4146-a0e8-3fc632ab20f5\") " pod="openshift-image-registry/node-ca-g2mdn" Oct 31 03:14:55.655006 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654567 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs\") pod \"network-metrics-daemon-lkp2n\" (UID: \"d680db8a-15e4-4817-bff1-a420f9f54681\") " pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:14:55.655006 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654600 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7a31b67-a1b9-4b6a-a112-69cd5c833019-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.655006 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654639 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/571b679b-fad2-423d-96e5-8d5550bb536d-iptables-alerter-script\") pod \"iptables-alerter-frjws\" (UID: \"571b679b-fad2-423d-96e5-8d5550bb536d\") " pod="openshift-network-operator/iptables-alerter-frjws" Oct 31 03:14:55.655006 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654672 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8b2g\" (UniqueName: \"kubernetes.io/projected/571b679b-fad2-423d-96e5-8d5550bb536d-kube-api-access-r8b2g\") pod \"iptables-alerter-frjws\" (UID: \"571b679b-fad2-423d-96e5-8d5550bb536d\") " pod="openshift-network-operator/iptables-alerter-frjws" Oct 31 03:14:55.655006 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654701 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-host\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.655006 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654728 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55f1a18a-59ef-4146-a0e8-3fc632ab20f5-host\") pod \"node-ca-g2mdn\" (UID: \"55f1a18a-59ef-4146-a0e8-3fc632ab20f5\") " pod="openshift-image-registry/node-ca-g2mdn" Oct 31 03:14:55.655006 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654771 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7a31b67-a1b9-4b6a-a112-69cd5c833019-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.655006 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654804 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-system-cni-dir\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.655006 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.654934 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.655489 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.655459 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/904f42d7-69e3-462f-9055-b6b840800f13-cni-binary-copy\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.655542 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.655491 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d877fe5-7b94-4645-bff5-6df0796bb637-kubelet-config\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:14:55.655542 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.655521 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-sysconfig\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.655611 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.655547 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-sys\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.655611 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.655576 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hnbq\" (UniqueName: \"kubernetes.io/projected/904f42d7-69e3-462f-9055-b6b840800f13-kube-api-access-7hnbq\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.655680 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.655617 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7a31b67-a1b9-4b6a-a112-69cd5c833019-system-cni-dir\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.655680 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.655657 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7a31b67-a1b9-4b6a-a112-69cd5c833019-cni-binary-copy\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.655745 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.655682 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-var-lib-cni-multus\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.655745 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.655711 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-hostroot\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.655962 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.655842 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7a31b67-a1b9-4b6a-a112-69cd5c833019-cnibin\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.655962 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.655876 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7a31b67-a1b9-4b6a-a112-69cd5c833019-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.656210 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.656187 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-kubernetes\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.656280 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.656245 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-sysctl-d\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.656335 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.656280 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-run\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.657114 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.657093 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:14:55.657222 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:55.657171 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:14:55.658653 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.658629 2569 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 03:14:55.679667 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.679635 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8gt4x" Oct 31 03:14:55.688683 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.688657 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8gt4x" Oct 31 03:14:55.739580 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.739541 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 03:14:55.739831 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.739592 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 03:14:55.740215 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.739943 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5s7vx\"" Oct 31 03:14:55.740215 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.739974 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 03:14:55.740215 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.740049 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 03:14:55.740215 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.740065 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fn5cc\"" Oct 31 03:14:55.740215 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.740092 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 03:14:55.740215 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.740116 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 03:14:55.740558 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.740359 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 03:14:55.741522 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.740931 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 03:14:55.741522 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.741374 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 03:14:55.741522 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.741455 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qg7r7\"" Oct 31 03:14:55.741873 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.741811 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 03:14:55.741873 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.741811 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 03:14:55.741873 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.741838 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 03:14:55.741873 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.741846 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 03:14:55.741873 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.741864 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 03:14:55.742236 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.742191 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 03:14:55.742236 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.742220 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gmjq8\"" Oct 31 03:14:55.742354 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.742261 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 03:14:55.742354 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.742306 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rtvr4\"" Oct 31 03:14:55.742354 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.742309 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 03:14:55.742354 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.742352 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:14:55.742766 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.742749 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nklpg\"" Oct 31 03:14:55.742927 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.742911 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 03:14:55.742993 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.742960 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 03:14:55.743046 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.743004 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fkjgm\"" Oct 31 03:14:55.743154 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.743127 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r6lm2\"" Oct 31 03:14:55.743257 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.743169 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 03:14:55.743257 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.743207 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 03:14:55.743596 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.743576 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 03:14:55.760785 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.757444 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16b948f8-e3da-4285-8342-27cbd2dbd141-hosts-file\") pod \"node-resolver-z74z2\" (UID: \"16b948f8-e3da-4285-8342-27cbd2dbd141\") " pod="openshift-dns/node-resolver-z74z2" Oct 31 03:14:55.760785 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.757541 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4ttf\" (UniqueName: \"kubernetes.io/projected/400b676a-a3a0-459d-9da6-8d30b2d3b497-kube-api-access-n4ttf\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.760785 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.757615 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7a31b67-a1b9-4b6a-a112-69cd5c833019-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.760785 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.757707 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-sysconfig\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.760785 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.757741 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hnbq\" (UniqueName: \"kubernetes.io/projected/904f42d7-69e3-462f-9055-b6b840800f13-kube-api-access-7hnbq\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.760785 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.757780 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-systemd-units\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.760785 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.757807 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7a31b67-a1b9-4b6a-a112-69cd5c833019-cnibin\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.760785 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.757832 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7a31b67-a1b9-4b6a-a112-69cd5c833019-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.760785 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.757858 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-sysctl-d\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.760785 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.757881 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-run\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.760785 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.757905 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkr86\" (UniqueName: \"kubernetes.io/projected/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-kube-api-access-dkr86\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.760785 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.757937 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-cni-bin\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.760785 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.757964 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-multus-cni-dir\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.760785 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.757988 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d877fe5-7b94-4645-bff5-6df0796bb637-dbus\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:14:55.760785 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758012 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-modprobe-d\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.760785 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758044 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-tmp\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.760785 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758071 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.762629 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758101 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-ovn-node-metrics-cert\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.762629 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758127 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16b948f8-e3da-4285-8342-27cbd2dbd141-tmp-dir\") pod \"node-resolver-z74z2\" (UID: \"16b948f8-e3da-4285-8342-27cbd2dbd141\") " pod="openshift-dns/node-resolver-z74z2" Oct 31 03:14:55.762629 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758156 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z82fk\" (UniqueName: \"kubernetes.io/projected/55f1a18a-59ef-4146-a0e8-3fc632ab20f5-kube-api-access-z82fk\") pod \"node-ca-g2mdn\" (UID: \"55f1a18a-59ef-4146-a0e8-3fc632ab20f5\") " pod="openshift-image-registry/node-ca-g2mdn" Oct 31 03:14:55.762629 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758182 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-socket-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.762629 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758205 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-registration-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.762629 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758230 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qsw4\" (UniqueName: \"kubernetes.io/projected/d7a31b67-a1b9-4b6a-a112-69cd5c833019-kube-api-access-2qsw4\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.762629 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758249 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:14:55.762629 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758275 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-sysctl-conf\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.762629 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758292 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-etc-openvswitch\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.762629 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758308 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-env-overrides\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.762629 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758326 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w65q\" (UniqueName: \"kubernetes.io/projected/d680db8a-15e4-4817-bff1-a420f9f54681-kube-api-access-6w65q\") pod \"network-metrics-daemon-lkp2n\" (UID: \"d680db8a-15e4-4817-bff1-a420f9f54681\") " pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:14:55.762629 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758350 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-run-netns\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.762629 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758377 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7a31b67-a1b9-4b6a-a112-69cd5c833019-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.762629 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758366 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-multus-conf-dir\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.762629 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758404 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-var-lib-kubelet\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.762629 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758427 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.763169 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758472 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs\") pod \"network-metrics-daemon-lkp2n\" (UID: \"d680db8a-15e4-4817-bff1-a420f9f54681\") " pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:14:55.763169 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758501 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-sysconfig\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.763169 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758501 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7a31b67-a1b9-4b6a-a112-69cd5c833019-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.763169 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758536 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/571b679b-fad2-423d-96e5-8d5550bb536d-iptables-alerter-script\") pod \"iptables-alerter-frjws\" (UID: \"571b679b-fad2-423d-96e5-8d5550bb536d\") " pod="openshift-network-operator/iptables-alerter-frjws" Oct 31 03:14:55.763169 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758559 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7a31b67-a1b9-4b6a-a112-69cd5c833019-cnibin\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.763169 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758566 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55f1a18a-59ef-4146-a0e8-3fc632ab20f5-host\") pod \"node-ca-g2mdn\" (UID: \"55f1a18a-59ef-4146-a0e8-3fc632ab20f5\") " pod="openshift-image-registry/node-ca-g2mdn" Oct 31 03:14:55.763169 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758595 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-kubelet\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.763169 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758623 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-slash\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.763169 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758648 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-ovnkube-script-lib\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.763169 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758676 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-sysctl-d\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.763169 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758682 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-device-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.763169 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758881 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-multus-cni-dir\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.763169 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758897 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d877fe5-7b94-4645-bff5-6df0796bb637-dbus\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:14:55.763169 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758930 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-run\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.763169 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758955 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-modprobe-d\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.763169 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.758969 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-var-lib-kubelet\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.763169 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:55.759166 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:14:55.763169 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.759209 2569 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 03:14:55.763855 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:55.759232 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs podName:d680db8a-15e4-4817-bff1-a420f9f54681 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:56.259211064 +0000 UTC m=+2.991644123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs") pod "network-metrics-daemon-lkp2n" (UID: "d680db8a-15e4-4817-bff1-a420f9f54681") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:14:55.763855 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:55.759304 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:14:55.763855 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:55.759345 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret podName:7d877fe5-7b94-4645-bff5-6df0796bb637 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:56.259331328 +0000 UTC m=+2.991764373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret") pod "global-pull-secret-syncer-gkf5m" (UID: "7d877fe5-7b94-4645-bff5-6df0796bb637") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:14:55.763855 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.759475 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-sysctl-conf\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.763855 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.759482 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7a31b67-a1b9-4b6a-a112-69cd5c833019-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.763855 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.759674 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-system-cni-dir\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.763855 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.759861 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-system-cni-dir\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.763855 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.759903 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-multus-conf-dir\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.763855 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.759943 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/904f42d7-69e3-462f-9055-b6b840800f13-cni-binary-copy\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.763855 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.759974 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d877fe5-7b94-4645-bff5-6df0796bb637-kubelet-config\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:14:55.763855 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760002 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-sys\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.763855 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760063 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-run-netns\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.763855 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760092 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwzkk\" (UniqueName: \"kubernetes.io/projected/16b948f8-e3da-4285-8342-27cbd2dbd141-kube-api-access-pwzkk\") pod \"node-resolver-z74z2\" (UID: \"16b948f8-e3da-4285-8342-27cbd2dbd141\") " pod="openshift-dns/node-resolver-z74z2" Oct 31 03:14:55.763855 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760121 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-sys-fs\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.763855 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760150 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7a31b67-a1b9-4b6a-a112-69cd5c833019-system-cni-dir\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.763855 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760165 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7a31b67-a1b9-4b6a-a112-69cd5c833019-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.764436 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760178 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7a31b67-a1b9-4b6a-a112-69cd5c833019-cni-binary-copy\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.764436 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760245 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-sys\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.764436 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760277 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-var-lib-cni-multus\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.764436 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760302 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-hostroot\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.764436 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760355 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-ovnkube-config\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.764436 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760384 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c47g\" (UniqueName: \"kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g\") pod \"network-check-target-4tzpt\" (UID: \"d92bdf4f-f9e4-424c-9c87-ec015574ff1a\") " pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:14:55.764436 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760411 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-kubernetes\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.764436 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760440 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-cni-netd\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.764436 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760470 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-multus-socket-dir-parent\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.764436 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760500 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-tuned\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.764436 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760527 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-run-openvswitch\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.764436 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760629 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/571b679b-fad2-423d-96e5-8d5550bb536d-iptables-alerter-script\") pod \"iptables-alerter-frjws\" (UID: \"571b679b-fad2-423d-96e5-8d5550bb536d\") " pod="openshift-network-operator/iptables-alerter-frjws" Oct 31 03:14:55.764436 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760683 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-log-socket\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.764436 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760703 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d877fe5-7b94-4645-bff5-6df0796bb637-kubelet-config\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:14:55.764436 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760712 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqks5\" (UniqueName: \"kubernetes.io/projected/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-kube-api-access-pqks5\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.764436 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760742 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-cnibin\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.764436 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760780 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-run-netns\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.765153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760789 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/904f42d7-69e3-462f-9055-b6b840800f13-multus-daemon-config\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.765153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760810 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55f1a18a-59ef-4146-a0e8-3fc632ab20f5-host\") pod \"node-ca-g2mdn\" (UID: \"55f1a18a-59ef-4146-a0e8-3fc632ab20f5\") " pod="openshift-image-registry/node-ca-g2mdn" Oct 31 03:14:55.765153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760816 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-lib-modules\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.765153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760838 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-run-systemd\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.765153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760848 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-var-lib-cni-multus\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.765153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760872 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f247bb3-6322-43c8-bfad-f27a41e26500-konnectivity-ca\") pod \"konnectivity-agent-5lds2\" (UID: \"8f247bb3-6322-43c8-bfad-f27a41e26500\") " pod="kube-system/konnectivity-agent-5lds2" Oct 31 03:14:55.765153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760744 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7a31b67-a1b9-4b6a-a112-69cd5c833019-system-cni-dir\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.765153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.760946 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-hostroot\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.765153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761001 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-kubernetes\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.765153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761051 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-multus-socket-dir-parent\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.765153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761202 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-etc-selinux\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.765153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761238 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-run-k8s-cni-cncf-io\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.765153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761243 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-cnibin\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.765153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761298 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-var-lib-kubelet\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.765153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761306 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-run-k8s-cni-cncf-io\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.765153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761353 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-etc-kubernetes\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.765153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761380 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/571b679b-fad2-423d-96e5-8d5550bb536d-host-slash\") pod \"iptables-alerter-frjws\" (UID: \"571b679b-fad2-423d-96e5-8d5550bb536d\") " pod="openshift-network-operator/iptables-alerter-frjws" Oct 31 03:14:55.765153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761376 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/904f42d7-69e3-462f-9055-b6b840800f13-cni-binary-copy\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.766014 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761401 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-lib-modules\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.766014 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761435 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/571b679b-fad2-423d-96e5-8d5550bb536d-host-slash\") pod \"iptables-alerter-frjws\" (UID: \"571b679b-fad2-423d-96e5-8d5550bb536d\") " pod="openshift-network-operator/iptables-alerter-frjws" Oct 31 03:14:55.766014 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761444 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/55f1a18a-59ef-4146-a0e8-3fc632ab20f5-serviceca\") pod \"node-ca-g2mdn\" (UID: \"55f1a18a-59ef-4146-a0e8-3fc632ab20f5\") " pod="openshift-image-registry/node-ca-g2mdn" Oct 31 03:14:55.766014 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761473 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-var-lib-kubelet\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.766014 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761492 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-etc-kubernetes\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.766014 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761511 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-run-ovn\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.766014 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761632 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f247bb3-6322-43c8-bfad-f27a41e26500-agent-certs\") pod \"konnectivity-agent-5lds2\" (UID: \"8f247bb3-6322-43c8-bfad-f27a41e26500\") " pod="kube-system/konnectivity-agent-5lds2" Oct 31 03:14:55.766014 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761669 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-var-lib-cni-bin\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.766014 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761707 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/904f42d7-69e3-462f-9055-b6b840800f13-multus-daemon-config\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.766014 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761737 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-run-multus-certs\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.766014 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761766 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-systemd\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.766014 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761784 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-var-lib-cni-bin\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.766014 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761830 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-run-multus-certs\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.766014 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761840 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-systemd\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.766014 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761947 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7a31b67-a1b9-4b6a-a112-69cd5c833019-cni-binary-copy\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.766014 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761982 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/55f1a18a-59ef-4146-a0e8-3fc632ab20f5-serviceca\") pod \"node-ca-g2mdn\" (UID: \"55f1a18a-59ef-4146-a0e8-3fc632ab20f5\") " pod="openshift-image-registry/node-ca-g2mdn" Oct 31 03:14:55.766014 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.761986 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-node-log\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.766014 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.762029 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8b2g\" (UniqueName: \"kubernetes.io/projected/571b679b-fad2-423d-96e5-8d5550bb536d-kube-api-access-r8b2g\") pod \"iptables-alerter-frjws\" (UID: \"571b679b-fad2-423d-96e5-8d5550bb536d\") " pod="openshift-network-operator/iptables-alerter-frjws" Oct 31 03:14:55.766863 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.762069 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-host\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.766863 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.762114 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-var-lib-openvswitch\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.766863 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.762125 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-host\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.766863 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.762145 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-run-ovn-kubernetes\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.766863 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.763493 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-tmp\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.766863 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.766655 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-tuned\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.773484 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.773453 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qsw4\" (UniqueName: \"kubernetes.io/projected/d7a31b67-a1b9-4b6a-a112-69cd5c833019-kube-api-access-2qsw4\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.774065 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.774023 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hnbq\" (UniqueName: \"kubernetes.io/projected/904f42d7-69e3-462f-9055-b6b840800f13-kube-api-access-7hnbq\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.774610 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.774588 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkr86\" (UniqueName: \"kubernetes.io/projected/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-kube-api-access-dkr86\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.775060 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.775035 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w65q\" (UniqueName: \"kubernetes.io/projected/d680db8a-15e4-4817-bff1-a420f9f54681-kube-api-access-6w65q\") pod \"network-metrics-daemon-lkp2n\" (UID: \"d680db8a-15e4-4817-bff1-a420f9f54681\") " pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:14:55.775395 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.775356 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z82fk\" (UniqueName: \"kubernetes.io/projected/55f1a18a-59ef-4146-a0e8-3fc632ab20f5-kube-api-access-z82fk\") pod \"node-ca-g2mdn\" (UID: \"55f1a18a-59ef-4146-a0e8-3fc632ab20f5\") " pod="openshift-image-registry/node-ca-g2mdn" Oct 31 03:14:55.775882 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.775859 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8b2g\" (UniqueName: \"kubernetes.io/projected/571b679b-fad2-423d-96e5-8d5550bb536d-kube-api-access-r8b2g\") pod \"iptables-alerter-frjws\" (UID: \"571b679b-fad2-423d-96e5-8d5550bb536d\") " pod="openshift-network-operator/iptables-alerter-frjws" Oct 31 03:14:55.785157 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.785092 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal" event={"ID":"fc792e750c2606b177de944a501bf339","Type":"ContainerStarted","Data":"9464aa9bf3b2aa43daa2a9323f4fe8a5b4c275c88cedd1e89755a98a3df10be7"} Oct 31 03:14:55.786556 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.786497 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" event={"ID":"9dcc8bbdd4365b42723f25cd49f22a1c","Type":"ContainerStarted","Data":"ab39d17504c1ed393c39b9ff8d73ce1229730f78105dc51229457d973aaf7a67"} Oct 31 03:14:55.863319 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863281 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.863490 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863349 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-kubelet\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.863490 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863378 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-slash\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.863490 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863405 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-ovnkube-script-lib\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.863490 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863430 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-device-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.863490 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863460 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-run-netns\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.863733 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863485 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwzkk\" (UniqueName: \"kubernetes.io/projected/16b948f8-e3da-4285-8342-27cbd2dbd141-kube-api-access-pwzkk\") pod \"node-resolver-z74z2\" (UID: \"16b948f8-e3da-4285-8342-27cbd2dbd141\") " pod="openshift-dns/node-resolver-z74z2" Oct 31 03:14:55.863733 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863547 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-sys-fs\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.863733 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863576 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-ovnkube-config\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.863733 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863602 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c47g\" (UniqueName: \"kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g\") pod \"network-check-target-4tzpt\" (UID: \"d92bdf4f-f9e4-424c-9c87-ec015574ff1a\") " pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:14:55.863733 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863629 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-cni-netd\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.863733 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863656 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-run-openvswitch\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.863733 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863685 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-log-socket\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.863733 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863709 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqks5\" (UniqueName: \"kubernetes.io/projected/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-kube-api-access-pqks5\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.864109 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863756 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-run-systemd\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.864109 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863783 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f247bb3-6322-43c8-bfad-f27a41e26500-konnectivity-ca\") pod \"konnectivity-agent-5lds2\" (UID: \"8f247bb3-6322-43c8-bfad-f27a41e26500\") " pod="kube-system/konnectivity-agent-5lds2" Oct 31 03:14:55.864109 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863808 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-etc-selinux\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.864109 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863842 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-run-ovn\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.864109 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863874 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f247bb3-6322-43c8-bfad-f27a41e26500-agent-certs\") pod \"konnectivity-agent-5lds2\" (UID: \"8f247bb3-6322-43c8-bfad-f27a41e26500\") " pod="kube-system/konnectivity-agent-5lds2" Oct 31 03:14:55.864109 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863908 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-node-log\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.864109 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863943 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-var-lib-openvswitch\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.864109 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.863978 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-run-ovn-kubernetes\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.864109 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.864003 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16b948f8-e3da-4285-8342-27cbd2dbd141-hosts-file\") pod \"node-resolver-z74z2\" (UID: \"16b948f8-e3da-4285-8342-27cbd2dbd141\") " pod="openshift-dns/node-resolver-z74z2" Oct 31 03:14:55.864109 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.864073 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4ttf\" (UniqueName: \"kubernetes.io/projected/400b676a-a3a0-459d-9da6-8d30b2d3b497-kube-api-access-n4ttf\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.864109 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.864110 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-systemd-units\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.864630 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.864141 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-cni-bin\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.864630 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.864174 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.864630 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.864205 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-ovn-node-metrics-cert\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.864630 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.864234 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16b948f8-e3da-4285-8342-27cbd2dbd141-tmp-dir\") pod \"node-resolver-z74z2\" (UID: \"16b948f8-e3da-4285-8342-27cbd2dbd141\") " pod="openshift-dns/node-resolver-z74z2" Oct 31 03:14:55.864630 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.864262 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-socket-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.864630 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.864293 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-registration-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.864630 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.864344 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-etc-openvswitch\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.864630 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.864379 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-env-overrides\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.865095 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.864989 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-env-overrides\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.865095 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.865077 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.865197 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.865121 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-kubelet\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.865197 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.865161 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-slash\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.866806 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.865550 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-ovnkube-script-lib\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.866806 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.865597 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-device-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.866806 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.865651 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-run-netns\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.866806 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.865832 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-node-log\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.866806 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.865896 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-sys-fs\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.866806 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.865954 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-var-lib-openvswitch\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.866806 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.865954 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-run-ovn-kubernetes\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.866806 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.866004 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16b948f8-e3da-4285-8342-27cbd2dbd141-hosts-file\") pod \"node-resolver-z74z2\" (UID: \"16b948f8-e3da-4285-8342-27cbd2dbd141\") " pod="openshift-dns/node-resolver-z74z2" Oct 31 03:14:55.866806 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.866263 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-systemd-units\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.866806 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.866313 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-cni-bin\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.866806 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.866379 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.866806 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.866553 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-ovnkube-config\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.867482 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.866834 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-cni-netd\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.867482 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.866882 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-run-openvswitch\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.867482 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.866940 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-log-socket\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.867482 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.867177 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-run-systemd\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.867798 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.867775 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f247bb3-6322-43c8-bfad-f27a41e26500-konnectivity-ca\") pod \"konnectivity-agent-5lds2\" (UID: \"8f247bb3-6322-43c8-bfad-f27a41e26500\") " pod="kube-system/konnectivity-agent-5lds2" Oct 31 03:14:55.867953 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.867910 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-etc-selinux\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.868019 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.867990 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-run-ovn\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.868387 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.868361 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-registration-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.868474 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.868405 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-etc-openvswitch\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.868918 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.868864 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-socket-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.869155 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.869135 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16b948f8-e3da-4285-8342-27cbd2dbd141-tmp-dir\") pod \"node-resolver-z74z2\" (UID: \"16b948f8-e3da-4285-8342-27cbd2dbd141\") " pod="openshift-dns/node-resolver-z74z2" Oct 31 03:14:55.870098 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.870055 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-ovn-node-metrics-cert\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.870895 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.870870 2569 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:14:55.875548 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.875523 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f247bb3-6322-43c8-bfad-f27a41e26500-agent-certs\") pod \"konnectivity-agent-5lds2\" (UID: \"8f247bb3-6322-43c8-bfad-f27a41e26500\") " pod="kube-system/konnectivity-agent-5lds2" Oct 31 03:14:55.876930 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:55.876904 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:14:55.877018 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:55.876934 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:14:55.877018 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:55.876949 2569 projected.go:196] Error preparing data for projected volume kube-api-access-7c47g for pod openshift-network-diagnostics/network-check-target-4tzpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:14:55.881785 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:55.877017 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g podName:d92bdf4f-f9e4-424c-9c87-ec015574ff1a nodeName:}" failed. No retries permitted until 2026-10-31 03:14:56.376995137 +0000 UTC m=+3.109428192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7c47g" (UniqueName: "kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g") pod "network-check-target-4tzpt" (UID: "d92bdf4f-f9e4-424c-9c87-ec015574ff1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:14:55.881785 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.879864 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4ttf\" (UniqueName: \"kubernetes.io/projected/400b676a-a3a0-459d-9da6-8d30b2d3b497-kube-api-access-n4ttf\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:55.881785 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.880577 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwzkk\" (UniqueName: \"kubernetes.io/projected/16b948f8-e3da-4285-8342-27cbd2dbd141-kube-api-access-pwzkk\") pod \"node-resolver-z74z2\" (UID: \"16b948f8-e3da-4285-8342-27cbd2dbd141\") " pod="openshift-dns/node-resolver-z74z2" Oct 31 03:14:55.884207 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.884183 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqks5\" (UniqueName: \"kubernetes.io/projected/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-kube-api-access-pqks5\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:55.948136 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.948097 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h9wtm" Oct 31 03:14:55.958592 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.958558 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-frjws" Oct 31 03:14:55.959408 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:55.959377 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904f42d7_69e3_462f_9055_b6b840800f13.slice/crio-9ea9fdda0d4f51060b0fd95750c35c3a585170ff17cd0767f6e759856bf67cb8 WatchSource:0}: Error finding container 9ea9fdda0d4f51060b0fd95750c35c3a585170ff17cd0767f6e759856bf67cb8: Status 404 returned error can't find the container with id 9ea9fdda0d4f51060b0fd95750c35c3a585170ff17cd0767f6e759856bf67cb8 Oct 31 03:14:55.969340 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.969308 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g2mdn" Oct 31 03:14:55.969781 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:55.969719 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod571b679b_fad2_423d_96e5_8d5550bb536d.slice/crio-f149510e7151cb1a40537997bd10ed25b77594f9c712d5dc4faf1502efbd1b71 WatchSource:0}: Error finding container f149510e7151cb1a40537997bd10ed25b77594f9c712d5dc4faf1502efbd1b71: Status 404 returned error can't find the container with id f149510e7151cb1a40537997bd10ed25b77594f9c712d5dc4faf1502efbd1b71 Oct 31 03:14:55.982480 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:55.982444 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f1a18a_59ef_4146_a0e8_3fc632ab20f5.slice/crio-01fb80cbee538275da581600b645f0018e78e8a5438ae2314c668976efc95d2f WatchSource:0}: Error finding container 01fb80cbee538275da581600b645f0018e78e8a5438ae2314c668976efc95d2f: Status 404 returned error can't find the container with id 01fb80cbee538275da581600b645f0018e78e8a5438ae2314c668976efc95d2f Oct 31 03:14:55.989936 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.989910 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:14:55.992982 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.992930 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:14:55.997577 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:55.997555 2569 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:14:56.004630 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:56.004596 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a31b67_a1b9_4b6a_a112_69cd5c833019.slice/crio-3c37acdd97b379138de5515cee74a2e79bb0dcb22b5a4ce180156ce646d44942 WatchSource:0}: Error finding container 3c37acdd97b379138de5515cee74a2e79bb0dcb22b5a4ce180156ce646d44942: Status 404 returned error can't find the container with id 3c37acdd97b379138de5515cee74a2e79bb0dcb22b5a4ce180156ce646d44942 Oct 31 03:14:56.007816 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:56.007790 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad1f3a7_eea4_4104_b3ff_e860da35bf9c.slice/crio-1343effb323d6d5be9ba2f2c1de0ff3b377ded7bf22646aedd851f1cea0439ad WatchSource:0}: Error finding container 1343effb323d6d5be9ba2f2c1de0ff3b377ded7bf22646aedd851f1cea0439ad: Status 404 returned error can't find the container with id 1343effb323d6d5be9ba2f2c1de0ff3b377ded7bf22646aedd851f1cea0439ad Oct 31 03:14:56.007816 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:56.007796 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z74z2" Oct 31 03:14:56.015309 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:56.015104 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:14:56.017658 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:56.017593 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5lds2" Oct 31 03:14:56.021192 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:56.021159 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b948f8_e3da_4285_8342_27cbd2dbd141.slice/crio-84059b6591690137309e98aa0e50941340cb9d7a1ce880e7e93223834fdb32e4 WatchSource:0}: Error finding container 84059b6591690137309e98aa0e50941340cb9d7a1ce880e7e93223834fdb32e4: Status 404 returned error can't find the container with id 84059b6591690137309e98aa0e50941340cb9d7a1ce880e7e93223834fdb32e4 Oct 31 03:14:56.028335 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:56.028310 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:14:56.028470 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:56.028446 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cf7508_2fc5_49a5_9e71_5d9c73bf200e.slice/crio-945bab7a9e73e6bbd58ee18321120992a5f33158baf662d521741b3343e8b265 WatchSource:0}: Error finding container 945bab7a9e73e6bbd58ee18321120992a5f33158baf662d521741b3343e8b265: Status 404 returned error can't find the container with id 945bab7a9e73e6bbd58ee18321120992a5f33158baf662d521741b3343e8b265 Oct 31 03:14:56.033286 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:56.033259 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f247bb3_6322_43c8_bfad_f27a41e26500.slice/crio-3829dc4f52ebb213c7ce11bfd34086fe601be80ee39a8c9a63e8b1a4c079c258 WatchSource:0}: Error finding container 3829dc4f52ebb213c7ce11bfd34086fe601be80ee39a8c9a63e8b1a4c079c258: Status 404 returned error can't find the container with id 3829dc4f52ebb213c7ce11bfd34086fe601be80ee39a8c9a63e8b1a4c079c258 Oct 31 03:14:56.040671 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:14:56.040638 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400b676a_a3a0_459d_9da6_8d30b2d3b497.slice/crio-09895ee45100c465e50e818c21642f320e820aa5cc3885d4a329c06f9496167c WatchSource:0}: Error finding container 09895ee45100c465e50e818c21642f320e820aa5cc3885d4a329c06f9496167c: Status 404 returned error can't find the container with id 09895ee45100c465e50e818c21642f320e820aa5cc3885d4a329c06f9496167c Oct 31 03:14:56.267432 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:56.267334 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:14:56.267432 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:56.267384 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs\") pod \"network-metrics-daemon-lkp2n\" (UID: \"d680db8a-15e4-4817-bff1-a420f9f54681\") " pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:14:56.267786 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:56.267518 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:14:56.267786 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:56.267577 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs podName:d680db8a-15e4-4817-bff1-a420f9f54681 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:57.26755856 +0000 UTC m=+3.999991622 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs") pod "network-metrics-daemon-lkp2n" (UID: "d680db8a-15e4-4817-bff1-a420f9f54681") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:14:56.267786 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:56.267653 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:14:56.267786 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:56.267690 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret podName:7d877fe5-7b94-4645-bff5-6df0796bb637 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:57.267681682 +0000 UTC m=+4.000114721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret") pod "global-pull-secret-syncer-gkf5m" (UID: "7d877fe5-7b94-4645-bff5-6df0796bb637") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:14:56.468857 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:56.468808 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c47g\" (UniqueName: \"kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g\") pod \"network-check-target-4tzpt\" (UID: \"d92bdf4f-f9e4-424c-9c87-ec015574ff1a\") " pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:14:56.469108 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:56.468981 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:14:56.469108 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:56.469002 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:14:56.469108 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:56.469014 2569 projected.go:196] Error preparing data for projected volume kube-api-access-7c47g for pod openshift-network-diagnostics/network-check-target-4tzpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:14:56.469108 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:56.469084 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g podName:d92bdf4f-f9e4-424c-9c87-ec015574ff1a nodeName:}" failed. No retries permitted until 2026-10-31 03:14:57.469063885 +0000 UTC m=+4.201496928 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7c47g" (UniqueName: "kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g") pod "network-check-target-4tzpt" (UID: "d92bdf4f-f9e4-424c-9c87-ec015574ff1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:14:56.692301 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:56.692031 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:09:56 +0000 UTC" deadline="2028-06-12 17:41:10.595313615 +0000 UTC" Oct 31 03:14:56.692301 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:56.692101 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14174h26m13.90321738s" Oct 31 03:14:56.780605 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:56.780056 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:14:56.780605 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:56.780190 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:14:56.802968 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:56.802491 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5lds2" event={"ID":"8f247bb3-6322-43c8-bfad-f27a41e26500","Type":"ContainerStarted","Data":"3829dc4f52ebb213c7ce11bfd34086fe601be80ee39a8c9a63e8b1a4c079c258"} Oct 31 03:14:56.816894 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:56.816851 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" event={"ID":"92cf7508-2fc5-49a5-9e71-5d9c73bf200e","Type":"ContainerStarted","Data":"945bab7a9e73e6bbd58ee18321120992a5f33158baf662d521741b3343e8b265"} Oct 31 03:14:56.828192 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:56.828146 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcfc2" event={"ID":"d7a31b67-a1b9-4b6a-a112-69cd5c833019","Type":"ContainerStarted","Data":"3c37acdd97b379138de5515cee74a2e79bb0dcb22b5a4ce180156ce646d44942"} Oct 31 03:14:56.831768 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:56.831728 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g2mdn" event={"ID":"55f1a18a-59ef-4146-a0e8-3fc632ab20f5","Type":"ContainerStarted","Data":"01fb80cbee538275da581600b645f0018e78e8a5438ae2314c668976efc95d2f"} Oct 31 03:14:56.834412 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:56.834381 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-frjws" event={"ID":"571b679b-fad2-423d-96e5-8d5550bb536d","Type":"ContainerStarted","Data":"f149510e7151cb1a40537997bd10ed25b77594f9c712d5dc4faf1502efbd1b71"} Oct 31 03:14:56.870799 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:56.863303 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" event={"ID":"400b676a-a3a0-459d-9da6-8d30b2d3b497","Type":"ContainerStarted","Data":"09895ee45100c465e50e818c21642f320e820aa5cc3885d4a329c06f9496167c"} Oct 31 03:14:56.870799 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:56.869859 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z74z2" event={"ID":"16b948f8-e3da-4285-8342-27cbd2dbd141","Type":"ContainerStarted","Data":"84059b6591690137309e98aa0e50941340cb9d7a1ce880e7e93223834fdb32e4"} Oct 31 03:14:56.881791 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:56.871732 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" event={"ID":"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c","Type":"ContainerStarted","Data":"1343effb323d6d5be9ba2f2c1de0ff3b377ded7bf22646aedd851f1cea0439ad"} Oct 31 03:14:56.881791 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:56.875559 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h9wtm" event={"ID":"904f42d7-69e3-462f-9055-b6b840800f13","Type":"ContainerStarted","Data":"9ea9fdda0d4f51060b0fd95750c35c3a585170ff17cd0767f6e759856bf67cb8"} Oct 31 03:14:57.275028 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:57.274962 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:14:57.275028 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:57.275026 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs\") pod \"network-metrics-daemon-lkp2n\" (UID: \"d680db8a-15e4-4817-bff1-a420f9f54681\") " pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:14:57.275292 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:57.275188 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:14:57.275292 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:57.275258 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs podName:d680db8a-15e4-4817-bff1-a420f9f54681 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:59.27523811 +0000 UTC m=+6.007671170 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs") pod "network-metrics-daemon-lkp2n" (UID: "d680db8a-15e4-4817-bff1-a420f9f54681") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:14:57.275759 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:57.275703 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:14:57.275844 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:57.275780 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret podName:7d877fe5-7b94-4645-bff5-6df0796bb637 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:59.27574709 +0000 UTC m=+6.008180152 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret") pod "global-pull-secret-syncer-gkf5m" (UID: "7d877fe5-7b94-4645-bff5-6df0796bb637") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:14:57.480217 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:57.480158 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c47g\" (UniqueName: \"kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g\") pod \"network-check-target-4tzpt\" (UID: \"d92bdf4f-f9e4-424c-9c87-ec015574ff1a\") " pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:14:57.480425 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:57.480337 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:14:57.480425 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:57.480356 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:14:57.480425 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:57.480371 2569 projected.go:196] Error preparing data for projected volume kube-api-access-7c47g for pod openshift-network-diagnostics/network-check-target-4tzpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:14:57.480594 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:57.480434 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g podName:d92bdf4f-f9e4-424c-9c87-ec015574ff1a nodeName:}" failed. No retries permitted until 2026-10-31 03:14:59.480413925 +0000 UTC m=+6.212846987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7c47g" (UniqueName: "kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g") pod "network-check-target-4tzpt" (UID: "d92bdf4f-f9e4-424c-9c87-ec015574ff1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:14:57.694519 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:57.694481 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:09:56 +0000 UTC" deadline="2028-04-21 03:39:55.88533225 +0000 UTC" Oct 31 03:14:57.694519 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:57.694520 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12912h24m58.190817325s" Oct 31 03:14:57.783792 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:57.782634 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:14:57.783792 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:57.782862 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:14:57.783792 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:57.782978 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:14:57.783792 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:57.783083 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:14:57.918972 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:57.918685 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal" event={"ID":"fc792e750c2606b177de944a501bf339","Type":"ContainerStarted","Data":"859d43288fd5c20f6be76c8cf240c6a2524de8216694875ea2cef8256195811c"} Oct 31 03:14:58.781066 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:58.781017 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:14:58.781624 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:58.781163 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:14:58.936183 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:58.936089 2569 generic.go:356] "Generic (PLEG): container finished" podID="9dcc8bbdd4365b42723f25cd49f22a1c" containerID="392fe58da98574e342eb491e1c359bbb7928bc4a5be06969278d16c476b7f327" exitCode=0 Oct 31 03:14:58.937198 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:58.937166 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" event={"ID":"9dcc8bbdd4365b42723f25cd49f22a1c","Type":"ContainerDied","Data":"392fe58da98574e342eb491e1c359bbb7928bc4a5be06969278d16c476b7f327"} Oct 31 03:14:58.956679 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:58.955718 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal" podStartSLOduration=3.955697366 podStartE2EDuration="3.955697366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:57.936069473 +0000 UTC m=+4.668502534" watchObservedRunningTime="2026-10-31 03:14:58.955697366 +0000 UTC m=+5.688130434" Oct 31 03:14:59.298810 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:59.298003 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:14:59.298810 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:59.298059 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs\") pod \"network-metrics-daemon-lkp2n\" (UID: \"d680db8a-15e4-4817-bff1-a420f9f54681\") " pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:14:59.298810 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:59.298238 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:14:59.298810 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:59.298305 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs podName:d680db8a-15e4-4817-bff1-a420f9f54681 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:03.298286391 +0000 UTC m=+10.030719434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs") pod "network-metrics-daemon-lkp2n" (UID: "d680db8a-15e4-4817-bff1-a420f9f54681") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:14:59.298810 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:59.298383 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:14:59.298810 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:59.298423 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret podName:7d877fe5-7b94-4645-bff5-6df0796bb637 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:03.298411293 +0000 UTC m=+10.030844336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret") pod "global-pull-secret-syncer-gkf5m" (UID: "7d877fe5-7b94-4645-bff5-6df0796bb637") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:14:59.499342 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:59.499299 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c47g\" (UniqueName: \"kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g\") pod \"network-check-target-4tzpt\" (UID: \"d92bdf4f-f9e4-424c-9c87-ec015574ff1a\") " pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:14:59.499582 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:59.499470 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:14:59.499582 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:59.499489 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:14:59.499582 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:59.499503 2569 projected.go:196] Error preparing data for projected volume kube-api-access-7c47g for pod openshift-network-diagnostics/network-check-target-4tzpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:14:59.499582 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:59.499565 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g podName:d92bdf4f-f9e4-424c-9c87-ec015574ff1a nodeName:}" failed. No retries permitted until 2026-10-31 03:15:03.499546239 +0000 UTC m=+10.231979294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7c47g" (UniqueName: "kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g") pod "network-check-target-4tzpt" (UID: "d92bdf4f-f9e4-424c-9c87-ec015574ff1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:14:59.782056 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:59.781977 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:14:59.782516 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:59.782173 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:14:59.782913 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:59.782892 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:14:59.783020 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:14:59.782999 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:14:59.952730 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:59.948493 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" event={"ID":"9dcc8bbdd4365b42723f25cd49f22a1c","Type":"ContainerStarted","Data":"6763aef3ce6951abf3c15c898b2ccade39d013de1abd8e370e797065dfb2eba3"} Oct 31 03:14:59.963463 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:59.963419 2569 generic.go:356] "Generic (PLEG): container finished" podID="d7a31b67-a1b9-4b6a-a112-69cd5c833019" containerID="712c2ede07b5be05256c54c6bdad4f5ff756eee935808477f12bfcf700177cc6" exitCode=0 Oct 31 03:14:59.963676 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:14:59.963481 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcfc2" event={"ID":"d7a31b67-a1b9-4b6a-a112-69cd5c833019","Type":"ContainerDied","Data":"712c2ede07b5be05256c54c6bdad4f5ff756eee935808477f12bfcf700177cc6"} Oct 31 03:15:00.001507 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:00.000561 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" podStartSLOduration=5.000541821 podStartE2EDuration="5.000541821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:59.972811287 +0000 UTC m=+6.705244366" watchObservedRunningTime="2026-10-31 03:15:00.000541821 +0000 UTC m=+6.732974916" Oct 31 03:15:00.781302 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:00.780733 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:00.781302 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:00.780892 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:01.783447 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:01.783405 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:01.783869 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:01.783571 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:01.784041 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:01.784020 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:01.784144 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:01.784124 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:01.987294 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:01.987251 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5lds2" event={"ID":"8f247bb3-6322-43c8-bfad-f27a41e26500","Type":"ContainerStarted","Data":"86d75c08ecd5493020526daadaf8a8ee9ee5987fb034c16e3a91392ef7ed9ada"} Oct 31 03:15:02.006457 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:02.006381 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5lds2" podStartSLOduration=3.208652646 podStartE2EDuration="8.006366036s" totalImagesPullingTime="4.79771339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:02.005239558 +0000 UTC m=+8.737672622" watchObservedRunningTime="2026-10-31 03:15:02.006366036 +0000 UTC m=+8.738799100" Oct 31 03:15:02.780802 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:02.780765 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:02.780984 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:02.780901 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:02.992450 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:02.992360 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g2mdn" event={"ID":"55f1a18a-59ef-4146-a0e8-3fc632ab20f5","Type":"ContainerStarted","Data":"0524dc834da76643569dbe1e53caf35f33f831a691314944661a021c41209d9c"} Oct 31 03:15:02.996082 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:02.996013 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" event={"ID":"400b676a-a3a0-459d-9da6-8d30b2d3b497","Type":"ContainerStarted","Data":"b682a964bae602e6a2f58ce20a0fb41c736e4b3b99d82dd82bced15005491005"} Oct 31 03:15:03.336983 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:03.336058 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:03.336983 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:03.336126 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs\") pod \"network-metrics-daemon-lkp2n\" (UID: \"d680db8a-15e4-4817-bff1-a420f9f54681\") " pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:03.336983 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:03.336320 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:15:03.336983 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:03.336397 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs podName:d680db8a-15e4-4817-bff1-a420f9f54681 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:11.336376436 +0000 UTC m=+18.068809496 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs") pod "network-metrics-daemon-lkp2n" (UID: "d680db8a-15e4-4817-bff1-a420f9f54681") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:15:03.336983 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:03.336844 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:15:03.336983 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:03.336901 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret podName:7d877fe5-7b94-4645-bff5-6df0796bb637 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:11.336884934 +0000 UTC m=+18.069317985 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret") pod "global-pull-secret-syncer-gkf5m" (UID: "7d877fe5-7b94-4645-bff5-6df0796bb637") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:15:03.538523 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:03.538480 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c47g\" (UniqueName: \"kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g\") pod \"network-check-target-4tzpt\" (UID: \"d92bdf4f-f9e4-424c-9c87-ec015574ff1a\") " pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:03.538729 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:03.538690 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:15:03.538806 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:03.538716 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:15:03.538806 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:03.538746 2569 projected.go:196] Error preparing data for projected volume kube-api-access-7c47g for pod openshift-network-diagnostics/network-check-target-4tzpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:15:03.538916 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:03.538813 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g podName:d92bdf4f-f9e4-424c-9c87-ec015574ff1a nodeName:}" failed. No retries permitted until 2026-10-31 03:15:11.538791336 +0000 UTC m=+18.271224379 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7c47g" (UniqueName: "kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g") pod "network-check-target-4tzpt" (UID: "d92bdf4f-f9e4-424c-9c87-ec015574ff1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:15:03.785566 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:03.784844 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:03.785566 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:03.784974 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:03.785566 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:03.785365 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:03.785566 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:03.785468 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:04.428532 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:04.428485 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5lds2" Oct 31 03:15:04.435953 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:04.435912 2569 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 03:15:04.736485 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:04.736117 2569 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:15:04.436158067Z","UUID":"7effbe4d-45f0-42b9-89fa-07aa00cc9832","Handler":null,"Name":"","Endpoint":""} Oct 31 03:15:04.738270 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:04.738225 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 03:15:04.738270 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:04.738256 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 03:15:04.780150 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:04.779995 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:04.780150 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:04.780114 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:05.003491 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:05.003236 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" event={"ID":"400b676a-a3a0-459d-9da6-8d30b2d3b497","Type":"ContainerStarted","Data":"28da1a37e8259cbebb74c16558157ef5c27a00697f17b8f4e820d8a31dfbff8b"} Oct 31 03:15:05.005986 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:05.005220 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z74z2" event={"ID":"16b948f8-e3da-4285-8342-27cbd2dbd141","Type":"ContainerStarted","Data":"be7285dbcbbef58137b72452a4043cb8c033b567bd761af9b8db0b1d70500d66"} Oct 31 03:15:05.008579 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:05.008549 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h9wtm" event={"ID":"904f42d7-69e3-462f-9055-b6b840800f13","Type":"ContainerStarted","Data":"d4e5a0f90ec67779de58c43965e3553030e2f3fc8733c5ad9743735ab09f354d"} Oct 31 03:15:05.014289 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:05.014259 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcfc2" event={"ID":"d7a31b67-a1b9-4b6a-a112-69cd5c833019","Type":"ContainerStarted","Data":"045e623de2d0c1e257011bb7c3e358f459f5a0efb2533fad29fc08173ccedd17"} Oct 31 03:15:05.023035 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:05.022982 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-z74z2" podStartSLOduration=2.255850585 podStartE2EDuration="11.022965483s" totalImagesPullingTime="8.767114898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:05.02236588 +0000 UTC m=+11.754798940" watchObservedRunningTime="2026-10-31 03:15:05.022965483 +0000 UTC m=+11.755398545" Oct 31 03:15:05.023159 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:05.023077 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g2mdn" podStartSLOduration=5.218031038 podStartE2EDuration="11.023072712s" totalImagesPullingTime="5.805041674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:03.011049014 +0000 UTC m=+9.743482082" watchObservedRunningTime="2026-10-31 03:15:05.023072712 +0000 UTC m=+11.755505775" Oct 31 03:15:05.060953 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:05.060891 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-h9wtm" podStartSLOduration=2.352312684 podStartE2EDuration="11.060870823s" totalImagesPullingTime="8.708558139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:05.040481682 +0000 UTC m=+11.772914744" watchObservedRunningTime="2026-10-31 03:15:05.060870823 +0000 UTC m=+11.793303886" Oct 31 03:15:05.780827 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:05.780796 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:05.781492 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:05.780932 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:05.781492 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:05.781022 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:05.783256 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:05.783225 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:06.019352 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:06.019311 2569 generic.go:356] "Generic (PLEG): container finished" podID="d7a31b67-a1b9-4b6a-a112-69cd5c833019" containerID="045e623de2d0c1e257011bb7c3e358f459f5a0efb2533fad29fc08173ccedd17" exitCode=0 Oct 31 03:15:06.019518 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:06.019404 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcfc2" event={"ID":"d7a31b67-a1b9-4b6a-a112-69cd5c833019","Type":"ContainerDied","Data":"045e623de2d0c1e257011bb7c3e358f459f5a0efb2533fad29fc08173ccedd17"} Oct 31 03:15:06.023657 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:06.023604 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" event={"ID":"400b676a-a3a0-459d-9da6-8d30b2d3b497","Type":"ContainerStarted","Data":"03ca0336063031917873e13ce5a00d38a38224138300c85d9fb075165a217ac3"} Oct 31 03:15:06.058356 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:06.058300 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" podStartSLOduration=3.072212788 podStartE2EDuration="12.058280797s" totalImagesPullingTime="8.986068009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:06.058040777 +0000 UTC m=+12.790473841" watchObservedRunningTime="2026-10-31 03:15:06.058280797 +0000 UTC m=+12.790713858" Oct 31 03:15:06.745962 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:06.745927 2569 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5lds2" Oct 31 03:15:06.746489 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:06.746467 2569 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5lds2" Oct 31 03:15:06.780188 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:06.780151 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:06.780365 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:06.780283 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:07.027933 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:07.027846 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-frjws" event={"ID":"571b679b-fad2-423d-96e5-8d5550bb536d","Type":"ContainerStarted","Data":"6f0d7fd0563b85e76a77a8b2f8cfd70f80009ff1490489f45c2528290774c036"} Oct 31 03:15:07.028656 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:07.028636 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5lds2" Oct 31 03:15:07.044232 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:07.044182 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-frjws" podStartSLOduration=4.348082481 podStartE2EDuration="13.04416824s" totalImagesPullingTime="8.696085759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:07.044161414 +0000 UTC m=+13.776594471" watchObservedRunningTime="2026-10-31 03:15:07.04416824 +0000 UTC m=+13.776601299" Oct 31 03:15:07.780160 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:07.780125 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:07.780353 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:07.780275 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:07.780353 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:07.780328 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:07.780468 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:07.780447 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:08.032495 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:08.032417 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" event={"ID":"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c","Type":"ContainerStarted","Data":"582bae619e24d3e93f9967d0e9d73d9734bb9d90b21ad1c0518042dcc52d98d9"} Oct 31 03:15:08.034918 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:08.034881 2569 generic.go:356] "Generic (PLEG): container finished" podID="d7a31b67-a1b9-4b6a-a112-69cd5c833019" containerID="287e7c4db900ea9c7194c278321a5f96e1d6f2f49685cd9c4cb4a72e1f526ba6" exitCode=0 Oct 31 03:15:08.035060 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:08.035011 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcfc2" event={"ID":"d7a31b67-a1b9-4b6a-a112-69cd5c833019","Type":"ContainerDied","Data":"287e7c4db900ea9c7194c278321a5f96e1d6f2f49685cd9c4cb4a72e1f526ba6"} Oct 31 03:15:08.062860 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:08.062802 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" podStartSLOduration=2.9905657249999997 podStartE2EDuration="14.062785829s" totalImagesPullingTime="11.072220104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:08.061915727 +0000 UTC m=+14.794348787" watchObservedRunningTime="2026-10-31 03:15:08.062785829 +0000 UTC m=+14.795218888" Oct 31 03:15:08.780643 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:08.780438 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:08.780819 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:08.780759 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:09.780742 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:09.780709 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:09.780742 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:09.780752 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:09.781279 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:09.780866 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:09.781279 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:09.780999 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:10.042363 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:10.042288 2569 generic.go:356] "Generic (PLEG): container finished" podID="d7a31b67-a1b9-4b6a-a112-69cd5c833019" containerID="848c30e377a3fac13afe5f53b4fa4aee586581e0ef72e05f19ca121d7726dc0a" exitCode=0 Oct 31 03:15:10.042363 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:10.042347 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcfc2" event={"ID":"d7a31b67-a1b9-4b6a-a112-69cd5c833019","Type":"ContainerDied","Data":"848c30e377a3fac13afe5f53b4fa4aee586581e0ef72e05f19ca121d7726dc0a"} Oct 31 03:15:10.780767 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:10.780732 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:10.781158 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:10.780865 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:11.400657 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:11.400613 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:11.400657 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:11.400665 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs\") pod \"network-metrics-daemon-lkp2n\" (UID: \"d680db8a-15e4-4817-bff1-a420f9f54681\") " pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:11.400923 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:11.400809 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:15:11.400923 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:11.400854 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:15:11.400923 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:11.400884 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret podName:7d877fe5-7b94-4645-bff5-6df0796bb637 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:27.400865052 +0000 UTC m=+34.133298097 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret") pod "global-pull-secret-syncer-gkf5m" (UID: "7d877fe5-7b94-4645-bff5-6df0796bb637") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:15:11.400923 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:11.400922 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs podName:d680db8a-15e4-4817-bff1-a420f9f54681 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:27.400905793 +0000 UTC m=+34.133338854 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs") pod "network-metrics-daemon-lkp2n" (UID: "d680db8a-15e4-4817-bff1-a420f9f54681") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:15:11.606846 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:11.606819 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c47g\" (UniqueName: \"kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g\") pod \"network-check-target-4tzpt\" (UID: \"d92bdf4f-f9e4-424c-9c87-ec015574ff1a\") " pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:11.607327 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:11.607307 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:15:11.607417 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:11.607331 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:15:11.607417 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:11.607347 2569 projected.go:196] Error preparing data for projected volume kube-api-access-7c47g for pod openshift-network-diagnostics/network-check-target-4tzpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:15:11.607417 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:11.607404 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g podName:d92bdf4f-f9e4-424c-9c87-ec015574ff1a nodeName:}" failed. No retries permitted until 2026-10-31 03:15:27.607384242 +0000 UTC m=+34.339817282 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7c47g" (UniqueName: "kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g") pod "network-check-target-4tzpt" (UID: "d92bdf4f-f9e4-424c-9c87-ec015574ff1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:15:11.781057 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:11.780895 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:11.781400 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:11.780967 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:11.781400 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:11.781133 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:11.781400 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:11.781242 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:12.050031 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:12.049986 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" event={"ID":"92cf7508-2fc5-49a5-9e71-5d9c73bf200e","Type":"ContainerStarted","Data":"b07e22250b502de55efe36178234914a59fd698ed9c2014f77548d4b3cf2c557"} Oct 31 03:15:12.050172 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:12.050093 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" event={"ID":"92cf7508-2fc5-49a5-9e71-5d9c73bf200e","Type":"ContainerStarted","Data":"b3369c203fd8ee92cc3090af77b8b61551b639b9190c1d115b16f6ac75fdbecb"} Oct 31 03:15:12.050172 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:12.050149 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" event={"ID":"92cf7508-2fc5-49a5-9e71-5d9c73bf200e","Type":"ContainerStarted","Data":"2c8dbc6423e9a021e18d52126b94b798f49f25de4bd45f0a9f8597c082f8b312"} Oct 31 03:15:12.050264 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:12.050200 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" event={"ID":"92cf7508-2fc5-49a5-9e71-5d9c73bf200e","Type":"ContainerStarted","Data":"42319638679ce87eba7d7090709ec3dad95a9295038e512b3eed38050a6dc98f"} Oct 31 03:15:12.050264 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:12.050248 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" event={"ID":"92cf7508-2fc5-49a5-9e71-5d9c73bf200e","Type":"ContainerStarted","Data":"706be7cbb0ec2c8c48a7698c5161da83d684ca302e110a637729ebb16152583c"} Oct 31 03:15:12.050347 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:12.050298 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" event={"ID":"92cf7508-2fc5-49a5-9e71-5d9c73bf200e","Type":"ContainerStarted","Data":"6a8cbd79da4d978759f9778cb256c9356739bcec94636301ae504bf744d543f3"} Oct 31 03:15:12.052847 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:12.052816 2569 generic.go:356] "Generic (PLEG): container finished" podID="d7a31b67-a1b9-4b6a-a112-69cd5c833019" containerID="a96dd3a582c40a0f95ea1ab2b4afdbcca86ab847366da1fe77a73f98a62d07a5" exitCode=0 Oct 31 03:15:12.052945 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:12.052876 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcfc2" event={"ID":"d7a31b67-a1b9-4b6a-a112-69cd5c833019","Type":"ContainerDied","Data":"a96dd3a582c40a0f95ea1ab2b4afdbcca86ab847366da1fe77a73f98a62d07a5"} Oct 31 03:15:12.780900 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:12.780857 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:12.781051 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:12.780971 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:13.058199 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:13.058121 2569 generic.go:356] "Generic (PLEG): container finished" podID="d7a31b67-a1b9-4b6a-a112-69cd5c833019" containerID="5dd6cb9c9a3f579ab659b40bf4d18ff6d7151502802997687df56bde8260c025" exitCode=0 Oct 31 03:15:13.058199 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:13.058172 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcfc2" event={"ID":"d7a31b67-a1b9-4b6a-a112-69cd5c833019","Type":"ContainerDied","Data":"5dd6cb9c9a3f579ab659b40bf4d18ff6d7151502802997687df56bde8260c025"} Oct 31 03:15:13.780832 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:13.780798 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:13.780980 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:13.780901 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:13.780980 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:13.780932 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:13.781071 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:13.780987 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:14.063494 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:14.063464 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zcfc2" event={"ID":"d7a31b67-a1b9-4b6a-a112-69cd5c833019","Type":"ContainerStarted","Data":"abbf376c8fe4fd1ca4aba70a07c7328d692e6caabc92fe85e9fa95e9ce4693ad"} Oct 31 03:15:14.086587 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:14.086539 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zcfc2" podStartSLOduration=8.209370715 podStartE2EDuration="20.08652246s" totalImagesPullingTime="11.877151745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:14.086138059 +0000 UTC m=+20.818571119" watchObservedRunningTime="2026-10-31 03:15:14.08652246 +0000 UTC m=+20.818955519" Oct 31 03:15:14.780403 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:14.780357 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:14.780563 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:14.780474 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:15.068142 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:15.068062 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" event={"ID":"92cf7508-2fc5-49a5-9e71-5d9c73bf200e","Type":"ContainerStarted","Data":"bf181c56726d6e0e42a26a4a90888fb802764c837e4b1f267c45bbea9e08a5f0"} Oct 31 03:15:15.785079 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:15.785050 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:15.785319 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:15.785050 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:15.785319 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:15.785179 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:15.785319 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:15.785210 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:16.103848 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:16.103756 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:16.301901 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:16.301708 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:16.780101 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:16.780069 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:16.780267 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:16.780164 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:17.075944 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:17.075867 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" event={"ID":"92cf7508-2fc5-49a5-9e71-5d9c73bf200e","Type":"ContainerStarted","Data":"a99a04b87073f70e205516382d11b25943f816bc997580e4731e76355f7cc864"} Oct 31 03:15:17.076113 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:17.076097 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:15:17.093108 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:17.093078 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:15:17.110037 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:17.109991 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" podStartSLOduration=7.5416244599999995 podStartE2EDuration="23.109979272s" totalImagesPullingTime="15.568354812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:17.109588309 +0000 UTC m=+23.842021369" watchObservedRunningTime="2026-10-31 03:15:17.109979272 +0000 UTC m=+23.842412331" Oct 31 03:15:17.558043 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:17.557962 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:17.696862 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:17.696821 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:17.780423 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:17.780392 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:17.780569 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:17.780438 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:17.780569 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:17.780539 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:17.780717 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:17.780696 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:17.905999 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:17.905922 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gkf5m"] Oct 31 03:15:17.908596 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:17.908568 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lkp2n"] Oct 31 03:15:17.909140 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:17.909118 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4tzpt"] Oct 31 03:15:17.909243 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:17.909218 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:17.909396 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:17.909347 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:18.078129 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:18.078102 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:18.078129 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:18.078110 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:18.078337 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:18.078200 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:18.078337 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:18.078256 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:18.078553 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:18.078540 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:15:18.078635 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:18.078612 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:15:18.093827 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:18.093802 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:15:19.780365 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:19.780322 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:19.780964 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:19.780322 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:19.780964 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:19.780439 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:19.780964 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:19.780325 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:19.780964 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:19.780510 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:19.780964 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:19.780579 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:20.046382 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:20.046291 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:20.178790 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:20.178758 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:21.035963 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:21.035936 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:21.177456 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:21.177426 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:21.305121 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:21.305080 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:21.449927 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:21.449844 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:21.599859 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:21.599820 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:21.780927 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:21.780844 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:21.780927 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:21.780870 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:21.780927 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:21.780870 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:21.781151 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:21.780946 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:21.781151 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:21.781035 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:21.781151 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:21.781102 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:22.062570 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:22.062458 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:22.235348 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:22.235316 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:22.745624 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:22.745422 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:23.747195 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:23.747163 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:23.783058 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:23.783025 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:23.783058 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:23.783051 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:23.783058 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:23.783065 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:23.783339 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:23.783169 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:23.783339 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:23.783279 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:23.783437 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:23.783352 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:23.888145 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:23.888109 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:24.045185 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:24.045087 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:24.125361 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:24.125332 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z74z2_16b948f8-e3da-4285-8342-27cbd2dbd141/dns-node-resolver/0.log" Oct 31 03:15:24.371248 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:24.371176 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:24.510404 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:24.510365 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:24.640259 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:24.640170 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:24.725341 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:24.725316 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g2mdn_55f1a18a-59ef-4146-a0e8-3fc632ab20f5/node-ca/0.log" Oct 31 03:15:25.175037 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:25.174988 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:25.325446 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:25.325409 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:25.464091 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:25.464013 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:25.761417 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:25.761326 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:25.780771 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:25.780743 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:25.780920 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:25.780809 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:25.780920 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:25.780810 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:25.781003 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:25.780912 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:25.781038 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:25.780997 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:25.781082 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:25.781068 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:25.969692 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:25.969645 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:26.123739 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:26.123648 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:26.558058 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:26.558030 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:26.895837 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:26.895766 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:27.154991 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:27.154950 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:27.360308 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:27.360280 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:27.420438 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:27.420365 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:27.420438 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:27.420407 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs\") pod \"network-metrics-daemon-lkp2n\" (UID: \"d680db8a-15e4-4817-bff1-a420f9f54681\") " pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:27.420607 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:27.420500 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 03:15:27.420607 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:27.420520 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:15:27.420607 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:27.420570 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret podName:7d877fe5-7b94-4645-bff5-6df0796bb637 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:59.420550413 +0000 UTC m=+66.152983457 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret") pod "global-pull-secret-syncer-gkf5m" (UID: "7d877fe5-7b94-4645-bff5-6df0796bb637") : object "kube-system"/"original-pull-secret" not registered Oct 31 03:15:27.420607 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:27.420589 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs podName:d680db8a-15e4-4817-bff1-a420f9f54681 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:59.420579627 +0000 UTC m=+66.153012679 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs") pod "network-metrics-daemon-lkp2n" (UID: "d680db8a-15e4-4817-bff1-a420f9f54681") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 03:15:27.621594 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:27.621559 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c47g\" (UniqueName: \"kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g\") pod \"network-check-target-4tzpt\" (UID: \"d92bdf4f-f9e4-424c-9c87-ec015574ff1a\") " pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:27.622001 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:27.621706 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 03:15:27.622001 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:27.621721 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 03:15:27.622001 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:27.621729 2569 projected.go:196] Error preparing data for projected volume kube-api-access-7c47g for pod openshift-network-diagnostics/network-check-target-4tzpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:15:27.622001 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:27.621781 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g podName:d92bdf4f-f9e4-424c-9c87-ec015574ff1a nodeName:}" failed. No retries permitted until 2026-10-31 03:15:59.621766595 +0000 UTC m=+66.354199646 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7c47g" (UniqueName: "kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g") pod "network-check-target-4tzpt" (UID: "d92bdf4f-f9e4-424c-9c87-ec015574ff1a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 03:15:27.756664 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:27.756591 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:27.780544 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:27.780518 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:27.780691 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:27.780519 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:27.780691 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:27.780629 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:27.780779 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:27.780525 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:27.780779 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:27.780706 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:27.780938 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:27.780821 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:27.885926 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:27.885889 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:28.010230 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:28.010135 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:28.554346 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:28.554304 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:29.159812 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:29.159781 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:29.323634 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:29.323605 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:29.465583 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:29.465495 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:29.759929 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:29.759842 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:29.780891 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:29.780863 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:29.781026 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:29.780894 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:29.781026 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:29.780965 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:29.781105 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:29.781046 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:29.781196 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:29.781174 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:29.781264 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:29.781247 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:29.967491 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:29.967449 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:30.099619 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:30.099542 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:30.259457 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:30.259428 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:30.550738 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:30.550701 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:31.158530 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:31.158494 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:31.359281 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:31.359247 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:31.753956 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:31.753919 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:31.780996 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:31.780964 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:31.781156 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:31.780966 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:31.781156 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:31.781065 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:31.781156 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:31.780964 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:31.781253 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:31.781134 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:31.781253 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:31.781239 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:31.957606 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:31.957565 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:32.555931 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:32.555901 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:33.159070 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:33.159027 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:33.353094 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:33.353060 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:33.772396 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:33.772362 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:33.781198 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:33.781173 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:33.781321 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:33.781288 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:33.781321 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:33.781295 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:33.781428 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:33.781404 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:33.781464 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:33.781435 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:33.781507 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:33.781493 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:33.898076 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:33.898037 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:34.018056 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:34.018024 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:34.355871 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:34.355838 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:34.548634 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:34.548592 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:35.004800 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:35.004774 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:35.126284 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:35.126252 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:35.783653 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:35.783623 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:35.783831 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:35.783733 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:35.783831 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:35.783747 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:35.783831 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:35.783825 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:35.783934 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:35.783821 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:35.783934 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:35.783913 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:37.780415 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:37.780379 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:37.780806 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:37.780480 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:37.780806 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:37.780487 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:37.780806 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:37.780568 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:37.780806 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:37.780579 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:37.780806 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:37.780700 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:38.229325 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:38.229294 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:38.360439 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:38.360403 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:38.491391 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:38.491303 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:38.618943 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:38.618909 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:38.752862 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:38.752779 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:39.780598 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:39.780568 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:39.781018 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:39.780568 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:39.781018 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:39.780690 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gkf5m" podUID="7d877fe5-7b94-4645-bff5-6df0796bb637" Oct 31 03:15:39.781018 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:39.780568 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:39.781018 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:39.780758 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" Oct 31 03:15:39.781018 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:39.780872 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lkp2n" podUID="d680db8a-15e4-4817-bff1-a420f9f54681" Oct 31 03:15:40.444434 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:40.444404 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:40.658888 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:40.658856 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:41.519350 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.519324 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeReady" Oct 31 03:15:41.519743 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.519442 2569 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 03:15:41.572702 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.572653 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vcg5q"] Oct 31 03:15:41.575727 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.575705 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zngp8"] Oct 31 03:15:41.575858 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.575803 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vcg5q" Oct 31 03:15:41.578613 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.578596 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:15:41.584289 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.584270 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 03:15:41.584412 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.584312 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 03:15:41.584477 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.584420 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 03:15:41.584535 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.584509 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 03:15:41.584588 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.584545 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dk5h8\"" Oct 31 03:15:41.585015 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.585001 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 03:15:41.589774 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.589742 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vcg5q"] Oct 31 03:15:41.592530 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.592506 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zngp8"] Oct 31 03:15:41.605504 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.605481 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vh2gt"] Oct 31 03:15:41.608204 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.608190 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vh2gt" Oct 31 03:15:41.615908 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.615889 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08da34c7-c69a-4eb9-848a-399cab14840e-crio-socket\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:15:41.616016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.615914 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08da34c7-c69a-4eb9-848a-399cab14840e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:15:41.616016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.615940 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08da34c7-c69a-4eb9-848a-399cab14840e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:15:41.616016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.615962 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/062e4d3d-4460-4412-93d3-22681a9a74e6-config-volume\") pod \"dns-default-vcg5q\" (UID: \"062e4d3d-4460-4412-93d3-22681a9a74e6\") " pod="openshift-dns/dns-default-vcg5q" Oct 31 03:15:41.616016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.615985 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgbnp\" (UniqueName: \"kubernetes.io/projected/062e4d3d-4460-4412-93d3-22681a9a74e6-kube-api-access-kgbnp\") pod \"dns-default-vcg5q\" (UID: \"062e4d3d-4460-4412-93d3-22681a9a74e6\") " pod="openshift-dns/dns-default-vcg5q" Oct 31 03:15:41.616016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.616011 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w596\" (UniqueName: \"kubernetes.io/projected/08da34c7-c69a-4eb9-848a-399cab14840e-kube-api-access-2w596\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:15:41.616218 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.616033 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50818995-4b65-457e-9b55-250b548e80d3-cert\") pod \"ingress-canary-vh2gt\" (UID: \"50818995-4b65-457e-9b55-250b548e80d3\") " pod="openshift-ingress-canary/ingress-canary-vh2gt" Oct 31 03:15:41.616218 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.616056 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8ncx\" (UniqueName: \"kubernetes.io/projected/50818995-4b65-457e-9b55-250b548e80d3-kube-api-access-b8ncx\") pod \"ingress-canary-vh2gt\" (UID: \"50818995-4b65-457e-9b55-250b548e80d3\") " pod="openshift-ingress-canary/ingress-canary-vh2gt" Oct 31 03:15:41.616218 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.616145 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08da34c7-c69a-4eb9-848a-399cab14840e-data-volume\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:15:41.616218 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.616170 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/062e4d3d-4460-4412-93d3-22681a9a74e6-metrics-tls\") pod \"dns-default-vcg5q\" (UID: \"062e4d3d-4460-4412-93d3-22681a9a74e6\") " pod="openshift-dns/dns-default-vcg5q" Oct 31 03:15:41.616218 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.616193 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062e4d3d-4460-4412-93d3-22681a9a74e6-tmp-dir\") pod \"dns-default-vcg5q\" (UID: \"062e4d3d-4460-4412-93d3-22681a9a74e6\") " pod="openshift-dns/dns-default-vcg5q" Oct 31 03:15:41.617386 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.617364 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vh2gt"] Oct 31 03:15:41.684443 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.684351 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 03:15:41.684443 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.684350 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 03:15:41.684443 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.684362 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 03:15:41.684443 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.684369 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 03:15:41.684699 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.684424 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zf8rk\"" Oct 31 03:15:41.684699 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.684511 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dct2q\"" Oct 31 03:15:41.717015 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.716989 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50818995-4b65-457e-9b55-250b548e80d3-cert\") pod \"ingress-canary-vh2gt\" (UID: \"50818995-4b65-457e-9b55-250b548e80d3\") " pod="openshift-ingress-canary/ingress-canary-vh2gt" Oct 31 03:15:41.717072 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.717020 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8ncx\" (UniqueName: \"kubernetes.io/projected/50818995-4b65-457e-9b55-250b548e80d3-kube-api-access-b8ncx\") pod \"ingress-canary-vh2gt\" (UID: \"50818995-4b65-457e-9b55-250b548e80d3\") " pod="openshift-ingress-canary/ingress-canary-vh2gt" Oct 31 03:15:41.717109 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.717070 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08da34c7-c69a-4eb9-848a-399cab14840e-data-volume\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:15:41.717109 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.717087 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/062e4d3d-4460-4412-93d3-22681a9a74e6-metrics-tls\") pod \"dns-default-vcg5q\" (UID: \"062e4d3d-4460-4412-93d3-22681a9a74e6\") " pod="openshift-dns/dns-default-vcg5q" Oct 31 03:15:41.717109 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.717104 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062e4d3d-4460-4412-93d3-22681a9a74e6-tmp-dir\") pod \"dns-default-vcg5q\" (UID: \"062e4d3d-4460-4412-93d3-22681a9a74e6\") " pod="openshift-dns/dns-default-vcg5q" Oct 31 03:15:41.717203 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.717123 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08da34c7-c69a-4eb9-848a-399cab14840e-crio-socket\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:15:41.717203 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.717147 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08da34c7-c69a-4eb9-848a-399cab14840e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:15:41.717203 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.717180 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08da34c7-c69a-4eb9-848a-399cab14840e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:15:41.717344 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.717302 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/062e4d3d-4460-4412-93d3-22681a9a74e6-config-volume\") pod \"dns-default-vcg5q\" (UID: \"062e4d3d-4460-4412-93d3-22681a9a74e6\") " pod="openshift-dns/dns-default-vcg5q" Oct 31 03:15:41.717344 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.717327 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbnp\" (UniqueName: \"kubernetes.io/projected/062e4d3d-4460-4412-93d3-22681a9a74e6-kube-api-access-kgbnp\") pod \"dns-default-vcg5q\" (UID: \"062e4d3d-4460-4412-93d3-22681a9a74e6\") " pod="openshift-dns/dns-default-vcg5q" Oct 31 03:15:41.717448 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.717340 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08da34c7-c69a-4eb9-848a-399cab14840e-crio-socket\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:15:41.717501 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.717460 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w596\" (UniqueName: \"kubernetes.io/projected/08da34c7-c69a-4eb9-848a-399cab14840e-kube-api-access-2w596\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:15:41.717571 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.717545 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062e4d3d-4460-4412-93d3-22681a9a74e6-tmp-dir\") pod \"dns-default-vcg5q\" (UID: \"062e4d3d-4460-4412-93d3-22681a9a74e6\") " pod="openshift-dns/dns-default-vcg5q" Oct 31 03:15:41.717635 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.717571 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08da34c7-c69a-4eb9-848a-399cab14840e-data-volume\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:15:41.718290 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.718094 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08da34c7-c69a-4eb9-848a-399cab14840e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:15:41.718290 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.718096 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/062e4d3d-4460-4412-93d3-22681a9a74e6-config-volume\") pod \"dns-default-vcg5q\" (UID: \"062e4d3d-4460-4412-93d3-22681a9a74e6\") " pod="openshift-dns/dns-default-vcg5q" Oct 31 03:15:41.720946 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.720920 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/062e4d3d-4460-4412-93d3-22681a9a74e6-metrics-tls\") pod \"dns-default-vcg5q\" (UID: \"062e4d3d-4460-4412-93d3-22681a9a74e6\") " pod="openshift-dns/dns-default-vcg5q" Oct 31 03:15:41.721040 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.720924 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08da34c7-c69a-4eb9-848a-399cab14840e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:15:41.721040 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.720986 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50818995-4b65-457e-9b55-250b548e80d3-cert\") pod \"ingress-canary-vh2gt\" (UID: \"50818995-4b65-457e-9b55-250b548e80d3\") " pod="openshift-ingress-canary/ingress-canary-vh2gt" Oct 31 03:15:41.726055 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.726031 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8ncx\" (UniqueName: \"kubernetes.io/projected/50818995-4b65-457e-9b55-250b548e80d3-kube-api-access-b8ncx\") pod \"ingress-canary-vh2gt\" (UID: \"50818995-4b65-457e-9b55-250b548e80d3\") " pod="openshift-ingress-canary/ingress-canary-vh2gt" Oct 31 03:15:41.726275 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.726254 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgbnp\" (UniqueName: \"kubernetes.io/projected/062e4d3d-4460-4412-93d3-22681a9a74e6-kube-api-access-kgbnp\") pod \"dns-default-vcg5q\" (UID: \"062e4d3d-4460-4412-93d3-22681a9a74e6\") " pod="openshift-dns/dns-default-vcg5q" Oct 31 03:15:41.726357 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.726260 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w596\" (UniqueName: \"kubernetes.io/projected/08da34c7-c69a-4eb9-848a-399cab14840e-kube-api-access-2w596\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:15:41.780609 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.780573 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:41.780777 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.780629 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:41.780777 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.780629 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:41.784311 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.784286 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 03:15:41.784442 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.784356 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 03:15:41.784527 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.784441 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 03:15:41.785334 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.785316 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 03:15:41.785544 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.785310 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d5pxc\"" Oct 31 03:15:41.788487 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.785711 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rfxnp\"" Oct 31 03:15:41.788487 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.785847 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k8hk8\"" Oct 31 03:15:41.891380 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.891347 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vcg5q" Oct 31 03:15:41.900453 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.900432 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:15:41.918303 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:41.918268 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vh2gt" Oct 31 03:15:42.186100 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:42.186067 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vcg5q"] Oct 31 03:15:42.189344 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:42.189317 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zngp8"] Oct 31 03:15:42.192277 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:15:42.192253 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08da34c7_c69a_4eb9_848a_399cab14840e.slice/crio-383c4ca43cffb50de3458004d549df509bcc5995b59f83b821a5e9036ea423eb WatchSource:0}: Error finding container 383c4ca43cffb50de3458004d549df509bcc5995b59f83b821a5e9036ea423eb: Status 404 returned error can't find the container with id 383c4ca43cffb50de3458004d549df509bcc5995b59f83b821a5e9036ea423eb Oct 31 03:15:42.220043 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:42.219920 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vh2gt"] Oct 31 03:15:42.223402 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:15:42.223378 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50818995_4b65_457e_9b55_250b548e80d3.slice/crio-0618767972e39b44ad2edae67be30a4b1b9c31bbe1a2be109a292f34c78d17e1 WatchSource:0}: Error finding container 0618767972e39b44ad2edae67be30a4b1b9c31bbe1a2be109a292f34c78d17e1: Status 404 returned error can't find the container with id 0618767972e39b44ad2edae67be30a4b1b9c31bbe1a2be109a292f34c78d17e1 Oct 31 03:15:43.131076 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.131037 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vh2gt" event={"ID":"50818995-4b65-457e-9b55-250b548e80d3","Type":"ContainerStarted","Data":"0618767972e39b44ad2edae67be30a4b1b9c31bbe1a2be109a292f34c78d17e1"} Oct 31 03:15:43.133034 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.133005 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zngp8" event={"ID":"08da34c7-c69a-4eb9-848a-399cab14840e","Type":"ContainerStarted","Data":"38a9977ed0da20b2bc1a10b652930735502ff4e8199dd6c6ab03240b698f01f4"} Oct 31 03:15:43.133178 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.133063 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zngp8" event={"ID":"08da34c7-c69a-4eb9-848a-399cab14840e","Type":"ContainerStarted","Data":"383c4ca43cffb50de3458004d549df509bcc5995b59f83b821a5e9036ea423eb"} Oct 31 03:15:43.134926 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.134903 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vcg5q" event={"ID":"062e4d3d-4460-4412-93d3-22681a9a74e6","Type":"ContainerStarted","Data":"33995e191d31be212d6ba7963974885157df6f40f818511415175c4c39f249c1"} Oct 31 03:15:43.179413 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.179371 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-968x8"] Oct 31 03:15:43.184178 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.184153 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:15:43.190327 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.190304 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 03:15:43.192187 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.192167 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 03:15:43.193348 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.193312 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 31 03:15:43.193348 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.193327 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-htgcs\"" Oct 31 03:15:43.193348 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.193328 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 31 03:15:43.197010 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.196941 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-968x8"] Oct 31 03:15:43.229967 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.229559 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:43.230851 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.230823 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:15:43.230975 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.230891 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:15:43.230975 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.230944 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:15:43.231078 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.230988 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8bkq\" (UniqueName: \"kubernetes.io/projected/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-kube-api-access-q8bkq\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:15:43.295109 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.295067 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 03:15:43.332415 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.331917 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:15:43.332415 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.331995 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:15:43.332415 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.332044 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:15:43.332415 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:43.332059 2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Oct 31 03:15:43.332415 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.332085 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bkq\" (UniqueName: \"kubernetes.io/projected/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-kube-api-access-q8bkq\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:15:43.332415 ip-10-0-142-76 kubenswrapper[2569]: E1031 03:15:43.332134 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-prometheus-operator-tls podName:cf66bb31-fee6-49fd-aaea-c8e8d8361c77 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:43.832113399 +0000 UTC m=+50.564546457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-prometheus-operator-tls") pod "prometheus-operator-9cd89b74c-968x8" (UID: "cf66bb31-fee6-49fd-aaea-c8e8d8361c77") : secret "prometheus-operator-tls" not found Oct 31 03:15:43.332916 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.332892 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:15:43.334959 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.334914 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:15:43.342561 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.342536 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8bkq\" (UniqueName: \"kubernetes.io/projected/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-kube-api-access-q8bkq\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:15:43.371388 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.371347 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:43.741093 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.741015 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:43.835399 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.835356 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:15:43.838281 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:43.838249 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:15:44.032649 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:44.032552 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:44.137352 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:44.137148 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:15:44.141767 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:44.141565 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zngp8" event={"ID":"08da34c7-c69a-4eb9-848a-399cab14840e","Type":"ContainerStarted","Data":"3f5655433e36e3646703f05731a0286e697b90ebe2e64c674185fd197cf78dc8"} Oct 31 03:15:44.174468 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:44.174434 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:44.432788 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:44.431693 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-968x8"] Oct 31 03:15:45.145065 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:45.145023 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" event={"ID":"cf66bb31-fee6-49fd-aaea-c8e8d8361c77","Type":"ContainerStarted","Data":"a31b7e522fd5db8b986c7fc5a759eee40e3a666bdc4e0c2c07f4fcdfc78868d8"} Oct 31 03:15:45.147517 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:45.147488 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zngp8" event={"ID":"08da34c7-c69a-4eb9-848a-399cab14840e","Type":"ContainerStarted","Data":"cfcc76b08afa1eb87d3dedfa1ab09840a6be4e413e9ac4758b279564e30f5bd5"} Oct 31 03:15:45.151615 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:45.151590 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vcg5q" event={"ID":"062e4d3d-4460-4412-93d3-22681a9a74e6","Type":"ContainerStarted","Data":"c65eb5565edf1d200173c74e17522c51684f5df0ba89a3bc776c93cc18e5ee09"} Oct 31 03:15:45.151770 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:45.151635 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vcg5q" event={"ID":"062e4d3d-4460-4412-93d3-22681a9a74e6","Type":"ContainerStarted","Data":"6d1dee93b5632e1925eb4ff0404db1a2d7222904a61d79fbb0f2e08822594139"} Oct 31 03:15:45.151847 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:45.151807 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vcg5q" Oct 31 03:15:45.166185 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:45.166101 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zngp8" podStartSLOduration=1.6373581019999999 podStartE2EDuration="4.166089058s" totalImagesPullingTime="2.528730956s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:15:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:45.165525653 +0000 UTC m=+51.897958733" watchObservedRunningTime="2026-10-31 03:15:45.166089058 +0000 UTC m=+51.898522118" Oct 31 03:15:45.183460 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:45.183408 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-vcg5q" podStartSLOduration=2.046033938 podStartE2EDuration="4.183390972s" totalImagesPullingTime="2.137357034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:45.182486139 +0000 UTC m=+51.914919219" watchObservedRunningTime="2026-10-31 03:15:45.183390972 +0000 UTC m=+51.915824031" Oct 31 03:15:46.160070 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:46.160029 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vh2gt" event={"ID":"50818995-4b65-457e-9b55-250b548e80d3","Type":"ContainerStarted","Data":"8b91490cd2d451b4c65088d39e5285aa6a4de14943e4958f1b0679c90888d403"} Oct 31 03:15:46.161823 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:46.161796 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" event={"ID":"cf66bb31-fee6-49fd-aaea-c8e8d8361c77","Type":"ContainerStarted","Data":"33b1e980104111aa75685865943446aab2b6dd01dc4a7e58c266e665819730ac"} Oct 31 03:15:46.161940 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:46.161847 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" event={"ID":"cf66bb31-fee6-49fd-aaea-c8e8d8361c77","Type":"ContainerStarted","Data":"4037c9975aacaf60ab8ae9c580fe60910a3597a5ad0038be6407a34169f64b8f"} Oct 31 03:15:46.177342 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:46.177302 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vh2gt" podStartSLOduration=2.190231888 podStartE2EDuration="5.177289865s" totalImagesPullingTime="2.987057977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:46.176256342 +0000 UTC m=+52.908689404" watchObservedRunningTime="2026-10-31 03:15:46.177289865 +0000 UTC m=+52.909722924" Oct 31 03:15:46.191086 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:46.191046 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:46.201102 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:46.201055 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" podStartSLOduration=1.763851862 podStartE2EDuration="3.201041381s" totalImagesPullingTime="1.437189519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:46.20038144 +0000 UTC m=+52.932814524" watchObservedRunningTime="2026-10-31 03:15:46.201041381 +0000 UTC m=+52.933474435" Oct 31 03:15:46.771215 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:46.771178 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:46.908201 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:46.908164 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:48.529780 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.529742 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:48.745046 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.745013 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:48.791837 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.791758 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mtd46"] Oct 31 03:15:48.795160 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.795138 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.858810 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.858781 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 03:15:48.870138 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.870110 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 03:15:48.870418 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.870399 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p2pq6\"" Oct 31 03:15:48.871616 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.871599 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/575aa82a-6d21-46c1-80ff-7f756f00c108-metrics-client-ca\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.871693 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.871636 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/575aa82a-6d21-46c1-80ff-7f756f00c108-root\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.871693 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.871659 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-textfile\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.871773 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.871748 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/575aa82a-6d21-46c1-80ff-7f756f00c108-sys\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.871812 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.871789 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-tls\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.871850 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.871814 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-wtmp\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.871850 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.871831 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-accelerators-collector-config\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.871960 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.871946 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.872004 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.871968 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg9j9\" (UniqueName: \"kubernetes.io/projected/575aa82a-6d21-46c1-80ff-7f756f00c108-kube-api-access-pg9j9\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.872515 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.872495 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 03:15:48.972777 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.972729 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/575aa82a-6d21-46c1-80ff-7f756f00c108-root\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.972777 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.972781 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-textfile\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.973039 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.972806 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/575aa82a-6d21-46c1-80ff-7f756f00c108-sys\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.973039 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.972829 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-tls\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.973039 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.972848 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-wtmp\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.973039 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.972852 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/575aa82a-6d21-46c1-80ff-7f756f00c108-root\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.973039 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.972872 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-accelerators-collector-config\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.973039 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.972937 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.973039 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.972971 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg9j9\" (UniqueName: \"kubernetes.io/projected/575aa82a-6d21-46c1-80ff-7f756f00c108-kube-api-access-pg9j9\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.973039 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.973030 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-wtmp\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.973397 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.973086 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/575aa82a-6d21-46c1-80ff-7f756f00c108-metrics-client-ca\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.973397 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.973106 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/575aa82a-6d21-46c1-80ff-7f756f00c108-sys\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.973397 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.973155 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-textfile\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.973584 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.973566 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-accelerators-collector-config\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.973634 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.973594 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/575aa82a-6d21-46c1-80ff-7f756f00c108-metrics-client-ca\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.975315 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.975295 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.975425 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.975374 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-tls\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:48.994720 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:48.994687 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg9j9\" (UniqueName: \"kubernetes.io/projected/575aa82a-6d21-46c1-80ff-7f756f00c108-kube-api-access-pg9j9\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:49.025916 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:49.025888 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:49.109481 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:49.109413 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:15:49.121360 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:15:49.121328 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575aa82a_6d21_46c1_80ff_7f756f00c108.slice/crio-31ec223e35e7af6752b1dbb7ca38857f538db7258f801f5630640b324945bb38 WatchSource:0}: Error finding container 31ec223e35e7af6752b1dbb7ca38857f538db7258f801f5630640b324945bb38: Status 404 returned error can't find the container with id 31ec223e35e7af6752b1dbb7ca38857f538db7258f801f5630640b324945bb38 Oct 31 03:15:49.175116 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:49.175078 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mtd46" event={"ID":"575aa82a-6d21-46c1-80ff-7f756f00c108","Type":"ContainerStarted","Data":"31ec223e35e7af6752b1dbb7ca38857f538db7258f801f5630640b324945bb38"} Oct 31 03:15:49.197992 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:49.197960 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:49.348218 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:49.348180 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:49.728564 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:49.727935 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:49.927533 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:49.927498 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:50.067924 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.067841 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:50.094509 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.094473 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:15:50.266758 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.266724 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:50.767735 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.767700 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-b949d9bf6-8b6xc"] Oct 31 03:15:50.771034 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.771018 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.784421 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.784393 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 03:15:50.784421 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.784393 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 03:15:50.784601 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.784526 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 03:15:50.784601 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.784538 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-nwvk2\"" Oct 31 03:15:50.784601 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.784545 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 03:15:50.784601 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.784543 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 03:15:50.784802 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.784612 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2mhh0emo4u5nu\"" Oct 31 03:15:50.785867 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.785844 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b949d9bf6-8b6xc"] Oct 31 03:15:50.787195 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.787172 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-tls\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.787289 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.787203 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.787289 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.787228 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-metrics-client-ca\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.787364 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.787300 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.787469 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.787451 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-tmp\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.787527 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.787505 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkbnh\" (UniqueName: \"kubernetes.io/projected/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-kube-api-access-vkbnh\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.787569 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.787556 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.787612 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.787604 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.787705 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.787664 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-grpc-tls\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.888254 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.888220 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-tls\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.888254 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.888256 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.888455 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.888277 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-metrics-client-ca\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.888489 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.888460 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.888565 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.888539 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-tmp\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.888718 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.888598 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkbnh\" (UniqueName: \"kubernetes.io/projected/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-kube-api-access-vkbnh\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.888718 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.888629 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.888718 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.888667 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.888718 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.888704 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-grpc-tls\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.888952 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.888927 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-tmp\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.889193 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.889172 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-metrics-client-ca\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.891189 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.891165 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.891418 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.891396 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-tls\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.891455 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.891422 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.891492 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.891422 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.891697 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.891657 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.891803 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.891787 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-grpc-tls\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:50.896924 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:50.896903 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkbnh\" (UniqueName: \"kubernetes.io/projected/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-kube-api-access-vkbnh\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:51.093369 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:51.093284 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:51.182625 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:51.182595 2569 generic.go:356] "Generic (PLEG): container finished" podID="575aa82a-6d21-46c1-80ff-7f756f00c108" containerID="b4959fbd23721abded557e3b3abdffee4d9625488608538b0b8730854c3b5e19" exitCode=0 Oct 31 03:15:51.182785 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:51.182652 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mtd46" event={"ID":"575aa82a-6d21-46c1-80ff-7f756f00c108","Type":"ContainerDied","Data":"b4959fbd23721abded557e3b3abdffee4d9625488608538b0b8730854c3b5e19"} Oct 31 03:15:51.374037 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:51.373972 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-b949d9bf6-8b6xc"] Oct 31 03:15:51.376671 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:15:51.376643 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd395aca5_b8ff_4da1_ad31_dd26e2427c2e.slice/crio-9f18bbf3213ffb7fe24c465bdcea01d198c6b008dadf0db37a1dfc112c2a5a3a WatchSource:0}: Error finding container 9f18bbf3213ffb7fe24c465bdcea01d198c6b008dadf0db37a1dfc112c2a5a3a: Status 404 returned error can't find the container with id 9f18bbf3213ffb7fe24c465bdcea01d198c6b008dadf0db37a1dfc112c2a5a3a Oct 31 03:15:51.568625 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:51.568593 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:51.768622 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:51.768591 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:51.917411 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:51.917363 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:52.188855 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:52.188813 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mtd46" event={"ID":"575aa82a-6d21-46c1-80ff-7f756f00c108","Type":"ContainerStarted","Data":"6087ce00ae526dfc6bb85b8d1e619794565543c5bb0e05289383b860521c9e6d"} Oct 31 03:15:52.189046 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:52.188876 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mtd46" event={"ID":"575aa82a-6d21-46c1-80ff-7f756f00c108","Type":"ContainerStarted","Data":"4233d9edfe20b3120e1da39e09edb78c89b3fbd7da1365af3d9673cef76c0f1b"} Oct 31 03:15:52.190147 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:52.190121 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" event={"ID":"d395aca5-b8ff-4da1-ad31-dd26e2427c2e","Type":"ContainerStarted","Data":"9f18bbf3213ffb7fe24c465bdcea01d198c6b008dadf0db37a1dfc112c2a5a3a"} Oct 31 03:15:52.211177 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:52.211125 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mtd46" podStartSLOduration=3.103421643 podStartE2EDuration="4.211108697s" totalImagesPullingTime="1.039982027s" totalInitContainerRuntime="67.705027ms" isStatefulPod=true podCreationTimestamp="2026-10-31 03:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:52.209505445 +0000 UTC m=+58.941938507" watchObservedRunningTime="2026-10-31 03:15:52.211108697 +0000 UTC m=+58.943541758" Oct 31 03:15:53.044955 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:53.044920 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:53.193421 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:53.187854 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:53.378185 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:53.378165 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8"] Oct 31 03:15:53.381425 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:53.381407 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" Oct 31 03:15:53.388264 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:53.388237 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 03:15:53.388654 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:53.388636 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-j2pvp\"" Oct 31 03:15:53.393273 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:53.393248 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8"] Oct 31 03:15:53.410975 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:53.410950 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq8hk\" (UniqueName: \"kubernetes.io/projected/8bae2714-9c1b-44db-a6ef-b0b008cf1fa2-kube-api-access-jq8hk\") pod \"monitoring-plugin-77f4f76594-5kbd8\" (UID: \"8bae2714-9c1b-44db-a6ef-b0b008cf1fa2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" Oct 31 03:15:53.411109 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:53.410986 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8bae2714-9c1b-44db-a6ef-b0b008cf1fa2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5kbd8\" (UID: \"8bae2714-9c1b-44db-a6ef-b0b008cf1fa2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" Oct 31 03:15:53.505365 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:53.505342 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:53.511858 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:53.511831 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8bae2714-9c1b-44db-a6ef-b0b008cf1fa2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5kbd8\" (UID: \"8bae2714-9c1b-44db-a6ef-b0b008cf1fa2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" Oct 31 03:15:53.511995 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:53.511956 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq8hk\" (UniqueName: \"kubernetes.io/projected/8bae2714-9c1b-44db-a6ef-b0b008cf1fa2-kube-api-access-jq8hk\") pod \"monitoring-plugin-77f4f76594-5kbd8\" (UID: \"8bae2714-9c1b-44db-a6ef-b0b008cf1fa2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" Oct 31 03:15:53.514174 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:53.514149 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8bae2714-9c1b-44db-a6ef-b0b008cf1fa2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5kbd8\" (UID: \"8bae2714-9c1b-44db-a6ef-b0b008cf1fa2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" Oct 31 03:15:53.521521 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:53.521490 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq8hk\" (UniqueName: \"kubernetes.io/projected/8bae2714-9c1b-44db-a6ef-b0b008cf1fa2-kube-api-access-jq8hk\") pod \"monitoring-plugin-77f4f76594-5kbd8\" (UID: \"8bae2714-9c1b-44db-a6ef-b0b008cf1fa2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" Oct 31 03:15:53.637223 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:53.637133 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:53.718359 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:53.718326 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-j2pvp\"" Oct 31 03:15:53.726462 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:53.726442 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" Oct 31 03:15:53.997481 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:53.997286 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8"] Oct 31 03:15:54.000164 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:15:54.000122 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bae2714_9c1b_44db_a6ef_b0b008cf1fa2.slice/crio-ffdadb7c0f31868d1a63d3ad83d08d3bbe287f99b99b656a23ba9a7f3d00cb81 WatchSource:0}: Error finding container ffdadb7c0f31868d1a63d3ad83d08d3bbe287f99b99b656a23ba9a7f3d00cb81: Status 404 returned error can't find the container with id ffdadb7c0f31868d1a63d3ad83d08d3bbe287f99b99b656a23ba9a7f3d00cb81 Oct 31 03:15:54.201962 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.201923 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" event={"ID":"8bae2714-9c1b-44db-a6ef-b0b008cf1fa2","Type":"ContainerStarted","Data":"ffdadb7c0f31868d1a63d3ad83d08d3bbe287f99b99b656a23ba9a7f3d00cb81"} Oct 31 03:15:54.204312 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.204280 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" event={"ID":"d395aca5-b8ff-4da1-ad31-dd26e2427c2e","Type":"ContainerStarted","Data":"fc9bae23c36c028ee0bcc4a8a92838b6fa29b466b8d167be4608924b9515197a"} Oct 31 03:15:54.204440 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.204363 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" event={"ID":"d395aca5-b8ff-4da1-ad31-dd26e2427c2e","Type":"ContainerStarted","Data":"e1ebe057a3e0ac787165e3f2423426c06641dbcf1d80621a6612e0872ed79dbd"} Oct 31 03:15:54.204440 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.204414 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" event={"ID":"d395aca5-b8ff-4da1-ad31-dd26e2427c2e","Type":"ContainerStarted","Data":"b99151a21c3c37f63bed106e642d0e2534de52d947e6dbc499bd99ffff51e725"} Oct 31 03:15:54.972504 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.972469 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 03:15:54.978197 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.978169 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:54.985714 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.985687 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 03:15:54.985916 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.985776 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 03:15:54.985916 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.985898 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 03:15:54.986025 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.985958 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 03:15:54.995068 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.994890 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 03:15:54.995068 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.994929 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 03:15:54.995068 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.994896 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 03:15:54.995068 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.995014 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 03:15:54.995068 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.995022 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 03:15:54.995068 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.995057 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 03:15:54.995839 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.995819 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 03:15:54.995961 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.995897 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 03:15:54.995961 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.995933 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a303dr7af9aru\"" Oct 31 03:15:54.996094 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.995977 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-p6hhn\"" Oct 31 03:15:54.996094 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.996047 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 03:15:54.996232 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.996194 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 03:15:54.997184 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:54.997161 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 03:15:55.002159 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.002135 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 03:15:55.025200 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.025090 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.025200 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.025131 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.025200 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.025163 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.025524 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.025261 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.025524 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.025293 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.025524 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.025333 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.025524 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.025454 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.025524 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.025495 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k6ph\" (UniqueName: \"kubernetes.io/projected/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-kube-api-access-6k6ph\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.025786 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.025567 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.025786 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.025726 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.025786 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.025769 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.025925 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.025801 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.025925 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.025829 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.025925 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.025864 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.025925 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.025891 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-config-out\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.025925 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.025914 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.026140 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.025938 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-web-config\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.026140 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.025974 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.026140 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.026008 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-config\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.026140 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.026030 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.027171 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.027140 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:55.126492 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.126452 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.126492 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.126505 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.126796 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.126642 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.126796 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.126729 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.126796 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.126755 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.126796 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.126788 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.127000 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.126831 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.127000 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.126862 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k6ph\" (UniqueName: \"kubernetes.io/projected/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-kube-api-access-6k6ph\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.127000 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.126895 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.127000 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.126920 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.127000 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.126938 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.127000 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.126955 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.127000 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.126975 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.127336 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.127005 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.127336 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.127029 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-config-out\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.127336 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.127050 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.127336 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.127068 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-web-config\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.127336 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.127088 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.127336 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.127110 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-config\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.127336 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.127124 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.127336 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.127187 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.128133 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.127552 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.128133 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.127808 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.128664 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.128329 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.128824 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.128803 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.129372 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.129344 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.132621 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.132599 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.134160 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.133761 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.134160 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.134122 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.134999 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.134897 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.134999 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.134960 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.135984 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.135539 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.135984 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.135935 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-web-config\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.137083 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.136277 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-config-out\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.137083 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.136551 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.137083 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.137045 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.137311 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.137295 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.137867 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.137843 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k6ph\" (UniqueName: \"kubernetes.io/projected/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-kube-api-access-6k6ph\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.138077 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.138056 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.138253 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.138234 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-config\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.165079 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.165048 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vcg5q" Oct 31 03:15:55.218252 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.218210 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" event={"ID":"d395aca5-b8ff-4da1-ad31-dd26e2427c2e","Type":"ContainerStarted","Data":"39f13982a52dad21e20b7199611e197f900d86c6955372491b5923200cee7758"} Oct 31 03:15:55.218741 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.218299 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" event={"ID":"d395aca5-b8ff-4da1-ad31-dd26e2427c2e","Type":"ContainerStarted","Data":"3be49e180cd8a8b482c9e91f137f31496b0128117c009167d816ed45ab282859"} Oct 31 03:15:55.218741 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.218341 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" event={"ID":"d395aca5-b8ff-4da1-ad31-dd26e2427c2e","Type":"ContainerStarted","Data":"18134c2d26c3fd40b59cdb129c247603bc2d62af22c489e37dda6e0b631a1795"} Oct 31 03:15:55.218741 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.218549 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:15:55.243486 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.243402 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podStartSLOduration=2.4606401079999998 podStartE2EDuration="5.243385656s" totalImagesPullingTime="2.782745548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:55.240197746 +0000 UTC m=+61.972630808" watchObservedRunningTime="2026-10-31 03:15:55.243385656 +0000 UTC m=+61.975818716" Oct 31 03:15:55.314595 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.314558 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:15:55.606714 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.606671 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 03:15:55.609980 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:15:55.609944 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b3277a_2966_4f41_99cf_1a1de0ba9d0a.slice/crio-eee132e0769005a7f31e3d3b4c1f929f8d96f2bc565566ef9b26c3b9bfaf298a WatchSource:0}: Error finding container eee132e0769005a7f31e3d3b4c1f929f8d96f2bc565566ef9b26c3b9bfaf298a: Status 404 returned error can't find the container with id eee132e0769005a7f31e3d3b4c1f929f8d96f2bc565566ef9b26c3b9bfaf298a Oct 31 03:15:55.965633 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:55.965597 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:56.091212 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:56.091171 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:56.224160 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:56.224064 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" event={"ID":"8bae2714-9c1b-44db-a6ef-b0b008cf1fa2","Type":"ContainerStarted","Data":"a257bd0dcd38528c8d34a5864d2174ebac787609c1b4f19e9f007ac17bc11c41"} Oct 31 03:15:56.224613 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:56.224310 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" Oct 31 03:15:56.226153 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:56.226123 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74b3277a-2966-4f41-99cf-1a1de0ba9d0a","Type":"ContainerStarted","Data":"eee132e0769005a7f31e3d3b4c1f929f8d96f2bc565566ef9b26c3b9bfaf298a"} Oct 31 03:15:56.229620 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:56.229596 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" Oct 31 03:15:56.240350 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:56.240307 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" podStartSLOduration=1.700799414 podStartE2EDuration="3.240289993s" totalImagesPullingTime="1.539490579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:56.239662148 +0000 UTC m=+62.972095227" watchObservedRunningTime="2026-10-31 03:15:56.240289993 +0000 UTC m=+62.972723055" Oct 31 03:15:57.230831 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:57.230802 2569 generic.go:356] "Generic (PLEG): container finished" podID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerID="945c8971b228a65db2916c2b8a4ab87372c601d5d632272b26611e95c031c386" exitCode=0 Oct 31 03:15:57.231237 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:57.230881 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74b3277a-2966-4f41-99cf-1a1de0ba9d0a","Type":"ContainerDied","Data":"945c8971b228a65db2916c2b8a4ab87372c601d5d632272b26611e95c031c386"} Oct 31 03:15:58.166068 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:58.166033 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:58.307016 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:58.306977 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:15:59.475943 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:59.475901 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs\") pod \"network-metrics-daemon-lkp2n\" (UID: \"d680db8a-15e4-4817-bff1-a420f9f54681\") " pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:59.476404 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:59.475999 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:59.479247 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:59.479222 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 03:15:59.480103 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:59.480079 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 03:15:59.489623 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:59.489598 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs\") pod \"network-metrics-daemon-lkp2n\" (UID: \"d680db8a-15e4-4817-bff1-a420f9f54681\") " pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:59.489802 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:59.489669 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:59.505888 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:59.505859 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k8hk8\"" Oct 31 03:15:59.508336 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:59.508303 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rfxnp\"" Oct 31 03:15:59.513415 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:59.513394 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:15:59.516629 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:59.516603 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:15:59.678651 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:59.678609 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c47g\" (UniqueName: \"kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g\") pod \"network-check-target-4tzpt\" (UID: \"d92bdf4f-f9e4-424c-9c87-ec015574ff1a\") " pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:59.681432 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:59.681401 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 03:15:59.691963 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:59.691912 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 03:15:59.703147 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:59.703116 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c47g\" (UniqueName: \"kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g\") pod \"network-check-target-4tzpt\" (UID: \"d92bdf4f-f9e4-424c-9c87-ec015574ff1a\") " pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:59.799283 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:59.799215 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d5pxc\"" Oct 31 03:15:59.801375 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:59.801247 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gkf5m"] Oct 31 03:15:59.808343 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:15:59.806887 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d877fe5_7b94_4645_bff5_6df0796bb637.slice/crio-e55134de16b156aa4d627cbd1bab62c88209475371be2f582b51a1c9c0e6ffe3 WatchSource:0}: Error finding container e55134de16b156aa4d627cbd1bab62c88209475371be2f582b51a1c9c0e6ffe3: Status 404 returned error can't find the container with id e55134de16b156aa4d627cbd1bab62c88209475371be2f582b51a1c9c0e6ffe3 Oct 31 03:15:59.808343 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:59.807038 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:15:59.808852 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:15:59.808829 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lkp2n"] Oct 31 03:15:59.811728 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:15:59.811698 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd680db8a_15e4_4817_bff1_a420f9f54681.slice/crio-997830a36967c65110d033f84f7ba67a8edfbc487e468d6219ebeebf6e36702e WatchSource:0}: Error finding container 997830a36967c65110d033f84f7ba67a8edfbc487e468d6219ebeebf6e36702e: Status 404 returned error can't find the container with id 997830a36967c65110d033f84f7ba67a8edfbc487e468d6219ebeebf6e36702e Oct 31 03:16:00.090432 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:00.090399 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4tzpt"] Oct 31 03:16:00.093097 ip-10-0-142-76 kubenswrapper[2569]: W1031 03:16:00.093067 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd92bdf4f_f9e4_424c_9c87_ec015574ff1a.slice/crio-c4bd5e56fe730a54bdb7262f4cc4570f330c25c484754253c6f2e2786da61949 WatchSource:0}: Error finding container c4bd5e56fe730a54bdb7262f4cc4570f330c25c484754253c6f2e2786da61949: Status 404 returned error can't find the container with id c4bd5e56fe730a54bdb7262f4cc4570f330c25c484754253c6f2e2786da61949 Oct 31 03:16:00.247754 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:00.247715 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gkf5m" event={"ID":"7d877fe5-7b94-4645-bff5-6df0796bb637","Type":"ContainerStarted","Data":"e55134de16b156aa4d627cbd1bab62c88209475371be2f582b51a1c9c0e6ffe3"} Oct 31 03:16:00.249174 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:00.249142 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lkp2n" event={"ID":"d680db8a-15e4-4817-bff1-a420f9f54681","Type":"ContainerStarted","Data":"997830a36967c65110d033f84f7ba67a8edfbc487e468d6219ebeebf6e36702e"} Oct 31 03:16:00.250549 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:00.250519 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4tzpt" event={"ID":"d92bdf4f-f9e4-424c-9c87-ec015574ff1a","Type":"ContainerStarted","Data":"c4bd5e56fe730a54bdb7262f4cc4570f330c25c484754253c6f2e2786da61949"} Oct 31 03:16:00.381310 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:00.381216 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:00.521734 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:00.521700 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:01.235109 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:01.235033 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:16:01.261418 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:01.261382 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74b3277a-2966-4f41-99cf-1a1de0ba9d0a","Type":"ContainerStarted","Data":"60b9ceab13d0559e2a1a78023b130560057961a4e0e000c4387f52fc81acff90"} Oct 31 03:16:01.261600 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:01.261468 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74b3277a-2966-4f41-99cf-1a1de0ba9d0a","Type":"ContainerStarted","Data":"1ef4fa90eee8f33863f63f8d99682f28211953f3423beb11b161cee799829e17"} Oct 31 03:16:01.261600 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:01.261507 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74b3277a-2966-4f41-99cf-1a1de0ba9d0a","Type":"ContainerStarted","Data":"70dc7ed0f447a3f58f9f85ab5b6592af61e08fe846c746d1ed4e6c152d509099"} Oct 31 03:16:01.261600 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:01.261543 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74b3277a-2966-4f41-99cf-1a1de0ba9d0a","Type":"ContainerStarted","Data":"e9b8514fdc995e8fcd0a477419c170cb1908c1e61f48e3f434e11052201af26a"} Oct 31 03:16:01.261600 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:01.261578 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74b3277a-2966-4f41-99cf-1a1de0ba9d0a","Type":"ContainerStarted","Data":"9af6916b470d852a952ad50fd11465a94577599a53e39f1372d4623509a94a9b"} Oct 31 03:16:01.261837 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:01.261614 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74b3277a-2966-4f41-99cf-1a1de0ba9d0a","Type":"ContainerStarted","Data":"f09ca5e1f3c887b5375b981e2890e138df11d33248e20b2b13bf9d75c52e5166"} Oct 31 03:16:01.315270 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:01.314289 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.004695141 podStartE2EDuration="7.314267013s" totalImagesPullingTime="4.290145055s" totalInitContainerRuntime="19.426817ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:01.313731042 +0000 UTC m=+68.046164126" watchObservedRunningTime="2026-10-31 03:16:01.314267013 +0000 UTC m=+68.046700074" Oct 31 03:16:02.222321 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:02.222276 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:02.273613 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:02.273514 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lkp2n" event={"ID":"d680db8a-15e4-4817-bff1-a420f9f54681","Type":"ContainerStarted","Data":"b2a6c65a53146f4c00750d97a2c15c5e0d23a9fe6615739062e463e279363d2f"} Oct 31 03:16:02.273613 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:02.273571 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lkp2n" event={"ID":"d680db8a-15e4-4817-bff1-a420f9f54681","Type":"ContainerStarted","Data":"55a75651bc82db934f72e7b17000ef56c0b65be7e17d4ef715647dbab7142b03"} Oct 31 03:16:02.297208 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:02.297145 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lkp2n" podStartSLOduration=66.713555349 podStartE2EDuration="1m8.297131505s" totalImagesPullingTime="1.583576156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:02.294825394 +0000 UTC m=+69.027258455" watchObservedRunningTime="2026-10-31 03:16:02.297131505 +0000 UTC m=+69.029564565" Oct 31 03:16:02.399825 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:02.399782 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:02.637986 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:02.637891 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:02.874694 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:02.874651 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:03.156165 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:03.156119 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:03.327917 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:03.327883 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:04.282479 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:04.282446 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4tzpt" event={"ID":"d92bdf4f-f9e4-424c-9c87-ec015574ff1a","Type":"ContainerStarted","Data":"4fd576fc7a5270c26e71f4068f0d979f47ca6e2c297bd7814bb5805190502eb0"} Oct 31 03:16:04.282630 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:04.282575 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:16:04.304391 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:04.304327 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4tzpt" podStartSLOduration=66.315892335 podStartE2EDuration="1m10.304305555s" totalImagesPullingTime="3.98841322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:04.302957193 +0000 UTC m=+71.035390254" watchObservedRunningTime="2026-10-31 03:16:04.304305555 +0000 UTC m=+71.036738617" Oct 31 03:16:05.032580 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:05.032535 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:05.183155 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:05.183098 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:05.287745 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:05.287638 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gkf5m" event={"ID":"7d877fe5-7b94-4645-bff5-6df0796bb637","Type":"ContainerStarted","Data":"af183becee9ebd050f135962af24f6eeef289da39fb8cf511a4dddff1e283e79"} Oct 31 03:16:05.312846 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:05.312795 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gkf5m" podStartSLOduration=65.977610272 podStartE2EDuration="1m11.312778943s" totalImagesPullingTime="5.335168671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:05.311556281 +0000 UTC m=+72.043989342" watchObservedRunningTime="2026-10-31 03:16:05.312778943 +0000 UTC m=+72.045212019" Oct 31 03:16:05.314777 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:05.314750 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:05.408705 ip-10-0-142-76 kubenswrapper[2569]: I1031 03:16:05.408561 2569 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:16:05.408704 ip-10-0-142-76 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 03:16:05.421078 ip-10-0-142-76 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 03:16:05.421246 ip-10-0-142-76 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 03:16:05.421296 ip-10-0-142-76 systemd[1]: kubelet.service: Consumed 4.239s CPU time. Oct 31 03:16:05.433169 ip-10-0-142-76 systemd[1]: Starting Kubernetes Kubelet... Oct 31 03:16:05.494518 ip-10-0-142-76 kubenswrapper[5490]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:16:05.494518 ip-10-0-142-76 kubenswrapper[5490]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 03:16:05.494518 ip-10-0-142-76 kubenswrapper[5490]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:16:05.494518 ip-10-0-142-76 kubenswrapper[5490]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 03:16:05.496659 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496571 5490 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:16:05.496725 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496661 5490 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:16:05.496725 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496667 5490 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:16:05.496725 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496671 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:16:05.496725 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496688 5490 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:16:05.496725 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496692 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:16:05.496725 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496695 5490 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:16:05.496725 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496697 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:16:05.496725 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496701 5490 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:16:05.496725 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496703 5490 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:16:05.496725 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496706 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:16:05.496725 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496709 5490 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:16:05.496725 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496712 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:16:05.496725 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496715 5490 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:16:05.496725 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496718 5490 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:16:05.496725 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496720 5490 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:16:05.496725 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496723 5490 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:16:05.496725 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496726 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:16:05.496725 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496729 5490 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:16:05.496725 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496732 5490 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:16:05.497196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496735 5490 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:16:05.497196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496744 5490 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:16:05.497196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496747 5490 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:16:05.497196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496750 5490 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:16:05.497196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496753 5490 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:16:05.497196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496755 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:16:05.497196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496758 5490 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:16:05.497196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496761 5490 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:16:05.497196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496763 5490 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:16:05.497196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496766 5490 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:16:05.497196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496769 5490 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:16:05.497196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496772 5490 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:16:05.497196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496774 5490 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:16:05.497196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496777 5490 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:16:05.497196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496780 5490 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:16:05.497196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496783 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:16:05.497196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496785 5490 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:16:05.497196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496788 5490 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:16:05.497196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496790 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:16:05.497196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496793 5490 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:16:05.497701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496796 5490 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:16:05.497701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496799 5490 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:16:05.497701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496801 5490 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:16:05.497701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496804 5490 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:16:05.497701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496808 5490 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:16:05.497701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496811 5490 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:16:05.497701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496813 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:16:05.497701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496816 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:16:05.497701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496819 5490 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:16:05.497701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496822 5490 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:16:05.497701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496824 5490 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:16:05.497701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496827 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:16:05.497701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496830 5490 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:16:05.497701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496833 5490 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:16:05.497701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496835 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:16:05.497701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496838 5490 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:16:05.497701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496841 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:16:05.497701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496844 5490 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:16:05.497701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496846 5490 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:16:05.497701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496849 5490 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:16:05.498188 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496852 5490 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:16:05.498188 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496854 5490 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:16:05.498188 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496857 5490 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:16:05.498188 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496860 5490 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:16:05.498188 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496862 5490 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:16:05.498188 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496872 5490 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:16:05.498188 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496876 5490 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:16:05.498188 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496879 5490 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:16:05.498188 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496883 5490 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:16:05.498188 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496887 5490 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:16:05.498188 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496889 5490 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:16:05.498188 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496892 5490 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:16:05.498188 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496895 5490 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:16:05.498188 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496898 5490 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:16:05.498188 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496901 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:16:05.498188 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496904 5490 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:16:05.498188 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496907 5490 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:16:05.498188 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496909 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:16:05.498188 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496912 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:16:05.498653 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496915 5490 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:16:05.498653 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496918 5490 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:16:05.498653 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496921 5490 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:16:05.498653 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496923 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:16:05.498653 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496926 5490 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:16:05.498653 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496929 5490 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:16:05.498653 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496932 5490 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:16:05.498653 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496934 5490 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:16:05.498653 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496937 5490 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:16:05.498653 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496940 5490 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:16:05.498653 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496942 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:16:05.498653 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496945 5490 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:16:05.498653 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496948 5490 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:16:05.498653 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496950 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:16:05.498653 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496953 5490 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:16:05.498653 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496956 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:16:05.498653 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496958 5490 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:16:05.498653 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496961 5490 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:16:05.498653 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496964 5490 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:16:05.498653 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496972 5490 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:16:05.499186 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496975 5490 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:16:05.499186 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496978 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:16:05.499186 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496980 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:16:05.499186 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496983 5490 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:16:05.499186 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496987 5490 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:16:05.499186 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496990 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:16:05.499186 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.496996 5490 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:16:05.499186 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497831 5490 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:16:05.499186 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497839 5490 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:16:05.499186 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497843 5490 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:16:05.499186 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497846 5490 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:16:05.499186 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497850 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:16:05.499186 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497853 5490 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:16:05.499186 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497856 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:16:05.499186 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497859 5490 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:16:05.499186 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497861 5490 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:16:05.499186 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497864 5490 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:16:05.499186 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497867 5490 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:16:05.499186 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497870 5490 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:16:05.499186 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497873 5490 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:16:05.499671 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497876 5490 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:16:05.499671 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497878 5490 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:16:05.499671 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497881 5490 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:16:05.499671 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497884 5490 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:16:05.499671 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497886 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:16:05.499671 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497890 5490 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:16:05.499671 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497894 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:16:05.499671 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497897 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:16:05.499671 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497900 5490 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:16:05.499671 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497904 5490 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:16:05.499671 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497907 5490 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:16:05.499671 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497910 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:16:05.499671 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497913 5490 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:16:05.499671 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497916 5490 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:16:05.499671 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497919 5490 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:16:05.499671 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497922 5490 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:16:05.499671 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497924 5490 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:16:05.499671 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497927 5490 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:16:05.499671 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497932 5490 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:16:05.499671 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497935 5490 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:16:05.500206 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497938 5490 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:16:05.500206 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497942 5490 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:16:05.500206 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497944 5490 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:16:05.500206 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497947 5490 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:16:05.500206 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497950 5490 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:16:05.500206 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497953 5490 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:16:05.500206 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497956 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:16:05.500206 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497958 5490 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:16:05.500206 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497961 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:16:05.500206 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497964 5490 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:16:05.500206 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497967 5490 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:16:05.500206 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497969 5490 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:16:05.500206 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497972 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:16:05.500206 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497975 5490 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:16:05.500206 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497977 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:16:05.500206 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497980 5490 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:16:05.500206 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497982 5490 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:16:05.500206 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497985 5490 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:16:05.500206 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497988 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:16:05.500206 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497990 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:16:05.500719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497993 5490 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:16:05.500719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497995 5490 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:16:05.500719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.497999 5490 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:16:05.500719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498001 5490 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:16:05.500719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498004 5490 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:16:05.500719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498007 5490 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:16:05.500719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498010 5490 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:16:05.500719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498012 5490 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:16:05.500719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498015 5490 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:16:05.500719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498017 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:16:05.500719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498021 5490 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:16:05.500719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498024 5490 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:16:05.500719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498027 5490 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:16:05.500719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498029 5490 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:16:05.500719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498032 5490 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:16:05.500719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498035 5490 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:16:05.500719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498037 5490 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:16:05.500719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498040 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:16:05.500719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498043 5490 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:16:05.500719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498045 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:16:05.501216 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498048 5490 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:16:05.501216 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498051 5490 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:16:05.501216 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498053 5490 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:16:05.501216 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498057 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:16:05.501216 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498060 5490 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:16:05.501216 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498062 5490 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:16:05.501216 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498065 5490 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:16:05.501216 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498067 5490 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:16:05.501216 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498070 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:16:05.501216 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498073 5490 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:16:05.501216 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498075 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:16:05.501216 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498078 5490 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:16:05.501216 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498081 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:16:05.501216 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498084 5490 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:16:05.501216 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498086 5490 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:16:05.501216 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498089 5490 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:16:05.501216 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498092 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:16:05.501216 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498095 5490 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:16:05.501216 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498099 5490 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:16:05.501752 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498102 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:16:05.501752 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498104 5490 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:16:05.501752 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498107 5490 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:16:05.501752 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498111 5490 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:16:05.501752 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498114 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:16:05.501752 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498117 5490 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:16:05.501752 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498120 5490 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:16:05.501752 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498123 5490 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:16:05.501752 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498128 5490 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:16:05.501752 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498132 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:16:05.501752 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498135 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:16:05.501752 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498139 5490 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:16:05.501752 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498142 5490 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:16:05.501752 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498145 5490 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:16:05.501752 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498649 5490 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:16:05.501752 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498656 5490 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:16:05.501752 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498661 5490 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:16:05.501752 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498665 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:16:05.501752 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498668 5490 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:16:05.502232 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498671 5490 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:16:05.502232 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498689 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:16:05.502232 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498692 5490 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:16:05.502232 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498695 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:16:05.502232 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498698 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:16:05.502232 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498701 5490 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:16:05.502232 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498704 5490 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:16:05.502232 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498707 5490 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:16:05.502232 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498709 5490 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:16:05.502232 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498712 5490 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:16:05.502232 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498715 5490 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:16:05.502232 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498717 5490 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:16:05.502232 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498721 5490 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:16:05.502232 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498724 5490 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:16:05.502232 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498726 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:16:05.502232 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498729 5490 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:16:05.502232 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498733 5490 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:16:05.502232 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498736 5490 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:16:05.502232 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498739 5490 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:16:05.502232 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498742 5490 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:16:05.502719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498745 5490 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:16:05.502719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498748 5490 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:16:05.502719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498750 5490 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:16:05.502719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498753 5490 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:16:05.502719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498755 5490 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:16:05.502719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498758 5490 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:16:05.502719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498761 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:16:05.502719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498763 5490 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:16:05.502719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498766 5490 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:16:05.502719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498769 5490 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:16:05.502719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498771 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:16:05.502719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498774 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:16:05.502719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498776 5490 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:16:05.502719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498779 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:16:05.502719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498781 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:16:05.502719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498784 5490 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:16:05.502719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498786 5490 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:16:05.502719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498789 5490 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:16:05.502719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498792 5490 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:16:05.502719 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498795 5490 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:16:05.503421 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498799 5490 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:16:05.503421 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498803 5490 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:16:05.503421 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498806 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:16:05.503421 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498809 5490 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:16:05.503421 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498812 5490 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:16:05.503421 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498816 5490 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:16:05.503421 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498818 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:16:05.503421 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498821 5490 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:16:05.503421 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498824 5490 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:16:05.503421 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498827 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:16:05.503421 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498829 5490 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:16:05.503421 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498835 5490 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:16:05.503421 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498837 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:16:05.503421 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498840 5490 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:16:05.503421 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498843 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:16:05.503421 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498846 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:16:05.503421 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498848 5490 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:16:05.503421 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498852 5490 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:16:05.503421 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498855 5490 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:16:05.504250 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498857 5490 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:16:05.504250 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498860 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:16:05.504250 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498863 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:16:05.504250 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498866 5490 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:16:05.504250 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498869 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:16:05.504250 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498872 5490 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:16:05.504250 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498874 5490 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:16:05.504250 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498877 5490 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:16:05.504250 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498879 5490 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:16:05.504250 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498882 5490 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:16:05.504250 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498884 5490 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:16:05.504250 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498887 5490 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:16:05.504250 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498890 5490 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:16:05.504250 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498893 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:16:05.504250 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498896 5490 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:16:05.504250 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498900 5490 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:16:05.504250 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498904 5490 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:16:05.504250 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498908 5490 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:16:05.504250 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498912 5490 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:16:05.504250 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498916 5490 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:16:05.505040 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498920 5490 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:16:05.505040 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498923 5490 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:16:05.505040 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498928 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:16:05.505040 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498932 5490 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:16:05.505040 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498938 5490 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:16:05.505040 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498942 5490 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:16:05.505040 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498947 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:16:05.505040 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498951 5490 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:16:05.505040 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498955 5490 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:16:05.505040 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498959 5490 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:16:05.505040 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498963 5490 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:16:05.505040 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498967 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:16:05.505040 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498972 5490 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:16:05.505040 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498976 5490 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:16:05.505040 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498990 5490 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:16:05.505040 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.498997 5490 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:16:05.505040 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499002 5490 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:16:05.505040 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499006 5490 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:16:05.505040 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499010 5490 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:16:05.505040 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499015 5490 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499019 5490 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499023 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499150 5490 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499159 5490 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499165 5490 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499170 5490 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499174 5490 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499178 5490 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499182 5490 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499187 5490 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499191 5490 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499194 5490 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499200 5490 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499204 5490 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499207 5490 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499210 5490 flags.go:64] FLAG: --cgroup-root="" Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499215 5490 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499220 5490 flags.go:64] FLAG: --client-ca-file="" Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499224 5490 flags.go:64] FLAG: --cloud-provider="external" Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499227 5490 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499231 5490 flags.go:64] FLAG: --cluster-domain="" Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499234 5490 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 03:16:05.505511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499238 5490 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499242 5490 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499245 5490 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499249 5490 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499253 5490 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499256 5490 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499259 5490 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499262 5490 flags.go:64] FLAG: --contention-profiling="false" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499266 5490 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499269 5490 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499272 5490 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499275 5490 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499280 5490 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499283 5490 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499286 5490 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499289 5490 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499293 5490 flags.go:64] FLAG: --enable-server="true" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499296 5490 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499300 5490 flags.go:64] FLAG: --event-burst="100" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499303 5490 flags.go:64] FLAG: --event-qps="50" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499307 5490 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499310 5490 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499313 5490 flags.go:64] FLAG: --eviction-hard="" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499318 5490 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 03:16:05.506095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499321 5490 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499325 5490 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499329 5490 flags.go:64] FLAG: --eviction-soft="" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499334 5490 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499337 5490 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499340 5490 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499343 5490 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499346 5490 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499350 5490 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499353 5490 flags.go:64] FLAG: --feature-gates="" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499357 5490 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499360 5490 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499364 5490 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499367 5490 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499370 5490 flags.go:64] FLAG: --healthz-port="10248" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499374 5490 flags.go:64] FLAG: --help="false" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499377 5490 flags.go:64] FLAG: --hostname-override="ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499380 5490 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499383 5490 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499386 5490 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499390 5490 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499397 5490 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499400 5490 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 03:16:05.506701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499403 5490 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499406 5490 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499409 5490 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499413 5490 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499416 5490 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499419 5490 flags.go:64] FLAG: --kube-reserved="" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499423 5490 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499426 5490 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499429 5490 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499432 5490 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499435 5490 flags.go:64] FLAG: --lock-file="" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499441 5490 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499445 5490 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499448 5490 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499455 5490 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499458 5490 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499462 5490 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499465 5490 flags.go:64] FLAG: --logging-format="text" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499468 5490 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499472 5490 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499475 5490 flags.go:64] FLAG: --manifest-url="" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499478 5490 flags.go:64] FLAG: --manifest-url-header="" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499483 5490 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499486 5490 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499490 5490 flags.go:64] FLAG: --max-pods="110" Oct 31 03:16:05.507257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499493 5490 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499496 5490 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499500 5490 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499503 5490 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499506 5490 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499509 5490 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499512 5490 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499520 5490 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499523 5490 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499526 5490 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499529 5490 flags.go:64] FLAG: --pod-cidr="" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499532 5490 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499536 5490 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499539 5490 flags.go:64] FLAG: --pods-per-core="0" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499542 5490 flags.go:64] FLAG: --port="10250" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499545 5490 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499548 5490 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0148485198cd11abf" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499552 5490 flags.go:64] FLAG: --qos-reserved="" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499556 5490 flags.go:64] FLAG: --read-only-port="10255" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499561 5490 flags.go:64] FLAG: --register-node="true" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499564 5490 flags.go:64] FLAG: --register-with-taints="" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499568 5490 flags.go:64] FLAG: --registry-burst="10" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499571 5490 flags.go:64] FLAG: --registry-qps="5" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499576 5490 flags.go:64] FLAG: --reserved-cpus="" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499580 5490 flags.go:64] FLAG: --reserved-memory="" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499586 5490 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 03:16:05.507864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499592 5490 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499597 5490 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499602 5490 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499607 5490 flags.go:64] FLAG: --runonce="false" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499610 5490 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499614 5490 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499617 5490 flags.go:64] FLAG: --seccomp-default="false" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499621 5490 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499624 5490 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499627 5490 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499630 5490 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499634 5490 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499637 5490 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499640 5490 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499643 5490 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499646 5490 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499649 5490 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499652 5490 flags.go:64] FLAG: --system-cgroups="" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499655 5490 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499661 5490 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499663 5490 flags.go:64] FLAG: --tls-cert-file="" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499666 5490 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499671 5490 flags.go:64] FLAG: --tls-min-version="" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499692 5490 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499697 5490 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 03:16:05.508505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499714 5490 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499717 5490 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499721 5490 flags.go:64] FLAG: --v="2" Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499726 5490 flags.go:64] FLAG: --version="false" Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499730 5490 flags.go:64] FLAG: --vmodule="" Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499734 5490 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.499738 5490 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499886 5490 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499890 5490 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499893 5490 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499896 5490 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499899 5490 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499901 5490 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499904 5490 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499907 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499909 5490 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499912 5490 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499915 5490 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499917 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499920 5490 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499923 5490 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:16:05.509222 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499926 5490 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:16:05.509743 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499929 5490 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:16:05.509743 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499931 5490 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:16:05.509743 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499934 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:16:05.509743 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499937 5490 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:16:05.509743 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499939 5490 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:16:05.509743 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499942 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:16:05.509743 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499945 5490 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:16:05.509743 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499947 5490 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:16:05.509743 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499950 5490 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:16:05.509743 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499953 5490 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:16:05.509743 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499959 5490 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:16:05.509743 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499961 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:16:05.509743 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499964 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:16:05.509743 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499967 5490 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:16:05.509743 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499970 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:16:05.509743 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499972 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:16:05.509743 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499975 5490 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:16:05.509743 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499978 5490 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:16:05.509743 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499981 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:16:05.510207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499984 5490 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:16:05.510207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499987 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:16:05.510207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499989 5490 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:16:05.510207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499992 5490 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:16:05.510207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499995 5490 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:16:05.510207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.499998 5490 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:16:05.510207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500000 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:16:05.510207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500003 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:16:05.510207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500006 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:16:05.510207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500008 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:16:05.510207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500011 5490 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:16:05.510207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500014 5490 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:16:05.510207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500017 5490 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:16:05.510207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500019 5490 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:16:05.510207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500022 5490 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:16:05.510207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500024 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:16:05.510207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500028 5490 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:16:05.510207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500033 5490 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:16:05.510207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500036 5490 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:16:05.510207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500040 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:16:05.510730 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500043 5490 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:16:05.510730 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500046 5490 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:16:05.510730 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500049 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:16:05.510730 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500055 5490 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:16:05.510730 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500058 5490 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:16:05.510730 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500061 5490 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:16:05.510730 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500064 5490 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:16:05.510730 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500067 5490 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:16:05.510730 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500070 5490 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:16:05.510730 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500072 5490 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:16:05.510730 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500075 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:16:05.510730 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500078 5490 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:16:05.510730 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500081 5490 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:16:05.510730 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500085 5490 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:16:05.510730 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500089 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:16:05.510730 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500092 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:16:05.510730 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500095 5490 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:16:05.510730 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500098 5490 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:16:05.510730 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500101 5490 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:16:05.511303 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500104 5490 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:16:05.511303 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500107 5490 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:16:05.511303 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500110 5490 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:16:05.511303 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500113 5490 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:16:05.511303 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500115 5490 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:16:05.511303 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500118 5490 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:16:05.511303 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500121 5490 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:16:05.511303 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500123 5490 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:16:05.511303 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500126 5490 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:16:05.511303 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500129 5490 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:16:05.511303 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500132 5490 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:16:05.511303 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500134 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:16:05.511303 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500137 5490 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:16:05.511303 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500139 5490 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:16:05.511303 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500142 5490 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:16:05.511303 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500145 5490 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:16:05.511303 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500150 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:16:05.511303 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500153 5490 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:16:05.511303 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500521 5490 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:16:05.511303 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500526 5490 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:16:05.511885 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500529 5490 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:16:05.511885 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500531 5490 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:16:05.511885 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500534 5490 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:16:05.511885 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500537 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:16:05.511885 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500539 5490 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:16:05.511885 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500542 5490 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:16:05.511885 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500545 5490 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:16:05.511885 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500548 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:16:05.511885 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500551 5490 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:16:05.511885 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500554 5490 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:16:05.511885 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500556 5490 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:16:05.511885 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500559 5490 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:16:05.511885 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500561 5490 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:16:05.511885 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.500594 5490 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:16:05.511885 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500630 5490 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:16:05.512309 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500635 5490 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:16:05.512309 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500639 5490 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:16:05.512309 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500642 5490 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:16:05.512309 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500645 5490 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:16:05.512309 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500648 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:16:05.512309 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500651 5490 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:16:05.512309 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500654 5490 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:16:05.512309 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500657 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:16:05.512309 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500659 5490 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:16:05.512309 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500662 5490 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:16:05.512309 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500665 5490 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:16:05.512309 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500667 5490 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:16:05.512309 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500670 5490 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:16:05.512309 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500686 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:16:05.512309 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500691 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:16:05.512309 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500694 5490 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:16:05.512309 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500697 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:16:05.512309 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500700 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:16:05.512309 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500703 5490 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:16:05.512794 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500706 5490 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:16:05.512794 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500709 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:16:05.512794 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500711 5490 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:16:05.512794 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500714 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:16:05.512794 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500717 5490 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:16:05.512794 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500720 5490 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:16:05.512794 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500723 5490 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:16:05.512794 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500726 5490 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:16:05.512794 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500728 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:16:05.512794 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500731 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:16:05.512794 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500734 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:16:05.512794 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500737 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:16:05.512794 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500740 5490 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:16:05.512794 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500742 5490 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:16:05.512794 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500745 5490 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:16:05.512794 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500748 5490 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:16:05.512794 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500751 5490 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:16:05.512794 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500753 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:16:05.512794 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500757 5490 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:16:05.512794 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500761 5490 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:16:05.513502 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500764 5490 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:16:05.513502 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500767 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:16:05.513502 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500769 5490 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:16:05.513502 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500772 5490 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:16:05.513502 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500775 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:16:05.513502 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500778 5490 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:16:05.513502 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500780 5490 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:16:05.513502 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500785 5490 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:16:05.513502 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500788 5490 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:16:05.513502 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500790 5490 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:16:05.513502 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500794 5490 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:16:05.513502 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500797 5490 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:16:05.513502 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500800 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:16:05.513502 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500802 5490 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:16:05.513502 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500805 5490 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:16:05.513502 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500810 5490 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:16:05.513502 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500814 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:16:05.513502 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500818 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:16:05.513502 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500821 5490 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:16:05.514215 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500823 5490 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:16:05.514215 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500826 5490 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:16:05.514215 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500829 5490 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:16:05.514215 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500832 5490 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:16:05.514215 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500835 5490 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:16:05.514215 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500838 5490 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:16:05.514215 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500840 5490 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:16:05.514215 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500843 5490 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:16:05.514215 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500846 5490 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:16:05.514215 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500848 5490 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:16:05.514215 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500852 5490 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:16:05.514215 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500854 5490 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:16:05.514215 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500857 5490 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:16:05.514215 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500860 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:16:05.514215 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500862 5490 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:16:05.514215 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500865 5490 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:16:05.514215 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500868 5490 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:16:05.514215 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500870 5490 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:16:05.514215 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500873 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:16:05.514215 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500876 5490 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:16:05.514735 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500881 5490 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:16:05.514735 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500884 5490 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:16:05.514735 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500887 5490 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:16:05.514735 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500889 5490 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:16:05.514735 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500893 5490 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:16:05.514735 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500895 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:16:05.514735 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500898 5490 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:16:05.514735 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500901 5490 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:16:05.514735 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500904 5490 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:16:05.514735 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500907 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:16:05.514735 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500910 5490 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:16:05.514735 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500912 5490 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:16:05.514735 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500915 5490 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:16:05.514735 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500918 5490 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:16:05.514735 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500920 5490 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:16:05.514735 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500924 5490 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:16:05.514735 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500927 5490 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:16:05.514735 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500930 5490 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:16:05.514735 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500933 5490 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:16:05.515241 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500935 5490 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:16:05.515241 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500938 5490 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:16:05.515241 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500941 5490 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:16:05.515241 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500943 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:16:05.515241 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500946 5490 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:16:05.515241 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500949 5490 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:16:05.515241 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500951 5490 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:16:05.515241 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.500954 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:16:05.515241 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.505890 5490 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 03:16:05.515241 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.505908 5490 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 03:16:05.515241 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.505927 5490 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 03:16:05.515241 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.505932 5490 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 03:16:05.515241 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.505989 5490 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:16:05.515241 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.505994 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:16:05.515241 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.505997 5490 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:16:05.515241 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506000 5490 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:16:05.515241 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506003 5490 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:16:05.515241 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506005 5490 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:16:05.515241 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506008 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:16:05.515241 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506011 5490 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:16:05.515751 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506014 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:16:05.515751 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506017 5490 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:16:05.515751 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506020 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:16:05.515751 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506022 5490 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:16:05.515751 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506026 5490 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:16:05.515751 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506031 5490 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:16:05.515751 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506035 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:16:05.515751 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506038 5490 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:16:05.515751 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506040 5490 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:16:05.515751 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506043 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:16:05.515751 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506046 5490 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:16:05.515751 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506049 5490 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:16:05.515751 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506051 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:16:05.515751 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506054 5490 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:16:05.515751 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506057 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:16:05.515751 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506060 5490 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:16:05.515751 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506063 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:16:05.515751 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506066 5490 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:16:05.515751 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506069 5490 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:16:05.516207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506072 5490 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:16:05.516207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506076 5490 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:16:05.516207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506079 5490 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:16:05.516207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506082 5490 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:16:05.516207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506085 5490 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:16:05.516207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506088 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:16:05.516207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506091 5490 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:16:05.516207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506094 5490 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:16:05.516207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506097 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:16:05.516207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506100 5490 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:16:05.516207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506103 5490 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:16:05.516207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506106 5490 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:16:05.516207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506109 5490 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:16:05.516207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506111 5490 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:16:05.516207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506114 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:16:05.516207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506117 5490 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:16:05.516207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506120 5490 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:16:05.516207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506122 5490 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:16:05.516207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506125 5490 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:16:05.516207 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506128 5490 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:16:05.516835 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506131 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:16:05.516835 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506134 5490 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:16:05.516835 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506136 5490 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:16:05.516835 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506140 5490 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:16:05.516835 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506143 5490 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:16:05.516835 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506145 5490 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:16:05.516835 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506148 5490 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:16:05.516835 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506151 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:16:05.516835 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506153 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:16:05.516835 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506156 5490 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:16:05.516835 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506158 5490 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:16:05.516835 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506162 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:16:05.516835 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506166 5490 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:16:05.516835 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506170 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:16:05.516835 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506174 5490 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:16:05.516835 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506177 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:16:05.516835 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506179 5490 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:16:05.516835 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506182 5490 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:16:05.516835 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506185 5490 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:16:05.517292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506213 5490 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:16:05.517292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506217 5490 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:16:05.517292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506220 5490 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:16:05.517292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506223 5490 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:16:05.517292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506225 5490 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:16:05.517292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506228 5490 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:16:05.517292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506231 5490 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:16:05.517292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506234 5490 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:16:05.517292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506236 5490 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:16:05.517292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506240 5490 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:16:05.517292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506242 5490 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:16:05.517292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506245 5490 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:16:05.517292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506248 5490 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:16:05.517292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506251 5490 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:16:05.517292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506254 5490 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:16:05.517292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506257 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:16:05.517292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506260 5490 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:16:05.517292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506262 5490 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:16:05.517292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506265 5490 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:16:05.517292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506268 5490 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:16:05.517918 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506271 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:16:05.517918 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506274 5490 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:16:05.517918 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506276 5490 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:16:05.517918 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506279 5490 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:16:05.517918 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506282 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:16:05.517918 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506285 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:16:05.517918 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506288 5490 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:16:05.517918 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506291 5490 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:16:05.517918 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506294 5490 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:16:05.517918 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506297 5490 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:16:05.517918 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506300 5490 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:16:05.517918 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506303 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:16:05.517918 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506305 5490 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:16:05.517918 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506308 5490 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:16:05.517918 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506310 5490 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:16:05.517918 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506313 5490 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:16:05.517918 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506316 5490 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:16:05.517918 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506318 5490 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:16:05.517918 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506321 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:16:05.518464 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506324 5490 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:16:05.518464 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.506353 5490 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:16:05.518464 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506493 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:16:05.518464 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506498 5490 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:16:05.518464 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506501 5490 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:16:05.518464 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506504 5490 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:16:05.518464 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506507 5490 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:16:05.518464 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506510 5490 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:16:05.518464 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506513 5490 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:16:05.518464 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506516 5490 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:16:05.518464 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506519 5490 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:16:05.518464 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506522 5490 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:16:05.518464 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506524 5490 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:16:05.518464 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506528 5490 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:16:05.518464 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506531 5490 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:16:05.518464 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506533 5490 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:16:05.518874 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506536 5490 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:16:05.518874 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506539 5490 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:16:05.518874 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506543 5490 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:16:05.518874 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506546 5490 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:16:05.518874 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506548 5490 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:16:05.518874 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506551 5490 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:16:05.518874 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506554 5490 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:16:05.518874 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506557 5490 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:16:05.518874 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506559 5490 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:16:05.518874 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506562 5490 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:16:05.518874 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506565 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:16:05.518874 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506567 5490 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:16:05.518874 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506570 5490 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:16:05.518874 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506574 5490 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:16:05.518874 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506578 5490 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:16:05.518874 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506582 5490 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:16:05.518874 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506585 5490 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:16:05.518874 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506588 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:16:05.518874 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506591 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:16:05.518874 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506594 5490 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:16:05.519344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506596 5490 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:16:05.519344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506599 5490 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:16:05.519344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506602 5490 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:16:05.519344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506605 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:16:05.519344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506608 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:16:05.519344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506610 5490 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:16:05.519344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506614 5490 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:16:05.519344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506616 5490 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:16:05.519344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506619 5490 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:16:05.519344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506622 5490 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:16:05.519344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506624 5490 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:16:05.519344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506627 5490 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:16:05.519344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506630 5490 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:16:05.519344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506633 5490 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:16:05.519344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506636 5490 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:16:05.519344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506639 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:16:05.519344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506641 5490 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:16:05.519344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506644 5490 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:16:05.519344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506647 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:16:05.519344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506650 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:16:05.519856 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506652 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:16:05.519856 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506655 5490 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:16:05.519856 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506658 5490 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:16:05.519856 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506661 5490 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:16:05.519856 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506664 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:16:05.519856 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506666 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:16:05.519856 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506669 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:16:05.519856 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506672 5490 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:16:05.519856 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506690 5490 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:16:05.519856 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506693 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:16:05.519856 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506696 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:16:05.519856 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506699 5490 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:16:05.519856 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506702 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:16:05.519856 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506705 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:16:05.519856 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506707 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:16:05.519856 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506710 5490 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:16:05.519856 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506715 5490 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:16:05.519856 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506719 5490 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:16:05.519856 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506723 5490 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:16:05.520311 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506726 5490 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:16:05.520311 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506728 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:16:05.520311 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506731 5490 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:16:05.520311 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506734 5490 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:16:05.520311 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506737 5490 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:16:05.520311 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506739 5490 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:16:05.520311 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506742 5490 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:16:05.520311 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506745 5490 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:16:05.520311 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506748 5490 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:16:05.520311 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506750 5490 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:16:05.520311 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506753 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:16:05.520311 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506756 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:16:05.520311 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506758 5490 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:16:05.520311 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506761 5490 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:16:05.520311 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506764 5490 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:16:05.520311 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506766 5490 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:16:05.520311 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506769 5490 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:16:05.520311 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506772 5490 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:16:05.520311 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506774 5490 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:16:05.520311 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506777 5490 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:16:05.521096 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506779 5490 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:16:05.521096 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506782 5490 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:16:05.521096 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506785 5490 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:16:05.521096 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506788 5490 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:16:05.521096 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506791 5490 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:16:05.521096 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506794 5490 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:16:05.521096 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506796 5490 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:16:05.521096 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506799 5490 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:16:05.521096 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506802 5490 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:16:05.521096 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506804 5490 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:16:05.521096 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506807 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:16:05.521096 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506810 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:16:05.521096 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506813 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:16:05.521096 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.506836 5490 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 03:16:05.521096 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506877 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 03:16:05.521096 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506882 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 03:16:05.521848 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506884 5490 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 03:16:05.521848 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506887 5490 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 03:16:05.521848 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506890 5490 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 03:16:05.521848 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506893 5490 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 03:16:05.521848 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506896 5490 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 03:16:05.521848 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506899 5490 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 03:16:05.521848 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506902 5490 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 03:16:05.521848 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506904 5490 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 03:16:05.521848 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506907 5490 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 03:16:05.521848 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506910 5490 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 03:16:05.521848 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506912 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 03:16:05.521848 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506915 5490 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 03:16:05.521848 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506920 5490 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 03:16:05.521848 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506923 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 03:16:05.521848 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506926 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 03:16:05.521848 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506929 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 03:16:05.521848 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506932 5490 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 03:16:05.521848 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506935 5490 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 03:16:05.521848 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506937 5490 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 03:16:05.521848 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506940 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 03:16:05.522337 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506943 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 03:16:05.522337 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506946 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 03:16:05.522337 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506949 5490 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 03:16:05.522337 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506952 5490 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 03:16:05.522337 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506955 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 03:16:05.522337 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506957 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 03:16:05.522337 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506960 5490 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 03:16:05.522337 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506963 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 03:16:05.522337 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506966 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 03:16:05.522337 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506969 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 03:16:05.522337 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506971 5490 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 03:16:05.522337 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506975 5490 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 03:16:05.522337 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506979 5490 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 03:16:05.522337 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506983 5490 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 03:16:05.522337 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506986 5490 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 03:16:05.522337 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506990 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 03:16:05.522337 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506993 5490 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 03:16:05.522337 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506996 5490 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 03:16:05.522799 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.506999 5490 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 03:16:05.522799 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507001 5490 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 03:16:05.522799 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507004 5490 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 03:16:05.522799 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507006 5490 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 03:16:05.522799 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507009 5490 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 03:16:05.522799 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507012 5490 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 03:16:05.522799 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507015 5490 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 03:16:05.522799 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507019 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 03:16:05.522799 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507021 5490 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 03:16:05.522799 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507024 5490 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 03:16:05.522799 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507027 5490 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 03:16:05.522799 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507029 5490 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 03:16:05.522799 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507032 5490 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 03:16:05.522799 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507035 5490 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 03:16:05.522799 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507038 5490 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 03:16:05.522799 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507040 5490 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 03:16:05.522799 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507043 5490 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 03:16:05.522799 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507046 5490 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 03:16:05.522799 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507049 5490 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 03:16:05.522799 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507052 5490 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 03:16:05.523312 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507054 5490 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 03:16:05.523312 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507058 5490 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 03:16:05.523312 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507061 5490 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 03:16:05.523312 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507064 5490 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 03:16:05.523312 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507067 5490 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 03:16:05.523312 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507070 5490 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 03:16:05.523312 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507073 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 03:16:05.523312 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507075 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 03:16:05.523312 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507078 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 03:16:05.523312 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507082 5490 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 03:16:05.523312 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507085 5490 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 03:16:05.523312 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507087 5490 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 03:16:05.523312 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507090 5490 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 03:16:05.523312 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507093 5490 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 03:16:05.523312 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507096 5490 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 03:16:05.523312 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507098 5490 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 03:16:05.523312 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507101 5490 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 03:16:05.523312 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507104 5490 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 03:16:05.523312 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507108 5490 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 03:16:05.523786 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507110 5490 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 03:16:05.523786 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507113 5490 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 03:16:05.523786 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507116 5490 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 03:16:05.523786 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507119 5490 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 03:16:05.523786 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507121 5490 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 03:16:05.523786 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507124 5490 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 03:16:05.523786 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507127 5490 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 03:16:05.523786 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507129 5490 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 03:16:05.523786 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507132 5490 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 03:16:05.523786 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507135 5490 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 03:16:05.523786 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507137 5490 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 03:16:05.523786 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507140 5490 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 03:16:05.523786 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507143 5490 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 03:16:05.523786 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507145 5490 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 03:16:05.523786 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507148 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 03:16:05.523786 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507151 5490 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 03:16:05.523786 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507153 5490 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 03:16:05.523786 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507157 5490 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 03:16:05.523786 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507161 5490 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 03:16:05.523786 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507164 5490 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 03:16:05.524273 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507167 5490 feature_gate.go:386] unrecognized feature gate: Example Oct 31 03:16:05.524273 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507170 5490 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 03:16:05.524273 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507175 5490 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 03:16:05.524273 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507177 5490 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 03:16:05.524273 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507180 5490 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 03:16:05.524273 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507183 5490 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 03:16:05.524273 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:05.507186 5490 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 03:16:05.524273 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.507247 5490 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 03:16:05.524273 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.508268 5490 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 03:16:05.524273 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.508326 5490 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 03:16:05.524273 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.508497 5490 server.go:1014] "Starting client certificate rotation" Oct 31 03:16:05.524273 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.508643 5490 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 03:16:05.524273 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.508705 5490 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:09:54 +0000 UTC" deadline="2028-07-09 08:38:01.342373494 +0000 UTC" Oct 31 03:16:05.524273 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.508734 5490 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14813h21m55.833641055s" Oct 31 03:16:05.524273 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.508983 5490 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:16:05.524273 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.510315 5490 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 03:16:05.524696 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.511459 5490 log.go:25] "Validated CRI v1 runtime API" Oct 31 03:16:05.524696 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.513258 5490 log.go:25] "Validated CRI v1 image API" Oct 31 03:16:05.524696 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.514016 5490 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 03:16:05.524696 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.516151 5490 fs.go:135] Filesystem UUIDs: map[2cd556e6-930f-4dae-9d56-d5647d119e3f:/dev/nvme0n1p3 65e4edfa-7282-429b-8d3b-c91b929ed9d6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Oct 31 03:16:05.524824 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.516171 5490 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01fb80cbee538275da581600b645f0018e78e8a5438ae2314c668976efc95d2f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01fb80cbee538275da581600b645f0018e78e8a5438ae2314c668976efc95d2f/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0618767972e39b44ad2edae67be30a4b1b9c31bbe1a2be109a292f34c78d17e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0618767972e39b44ad2edae67be30a4b1b9c31bbe1a2be109a292f34c78d17e1/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/09895ee45100c465e50e818c21642f320e820aa5cc3885d4a329c06f9496167c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/09895ee45100c465e50e818c21642f320e820aa5cc3885d4a329c06f9496167c/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31ec223e35e7af6752b1dbb7ca38857f538db7258f801f5630640b324945bb38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31ec223e35e7af6752b1dbb7ca38857f538db7258f801f5630640b324945bb38/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33995e191d31be212d6ba7963974885157df6f40f818511415175c4c39f249c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33995e191d31be212d6ba7963974885157df6f40f818511415175c4c39f249c1/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3829dc4f52ebb213c7ce11bfd34086fe601be80ee39a8c9a63e8b1a4c079c258/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3829dc4f52ebb213c7ce11bfd34086fe601be80ee39a8c9a63e8b1a4c079c258/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/383c4ca43cffb50de3458004d549df509bcc5995b59f83b821a5e9036ea423eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/383c4ca43cffb50de3458004d549df509bcc5995b59f83b821a5e9036ea423eb/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c37acdd97b379138de5515cee74a2e79bb0dcb22b5a4ce180156ce646d44942/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c37acdd97b379138de5515cee74a2e79bb0dcb22b5a4ce180156ce646d44942/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84059b6591690137309e98aa0e50941340cb9d7a1ce880e7e93223834fdb32e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84059b6591690137309e98aa0e50941340cb9d7a1ce880e7e93223834fdb32e4/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/945bab7a9e73e6bbd58ee18321120992a5f33158baf662d521741b3343e8b265/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/945bab7a9e73e6bbd58ee18321120992a5f33158baf662d521741b3343e8b265/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9464aa9bf3b2aa43daa2a9323f4fe8a5b4c275c88cedd1e89755a98a3df10be7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9464aa9bf3b2aa43daa2a9323f4fe8a5b4c275c88cedd1e89755a98a3df10be7/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/997830a36967c65110d033f84f7ba67a8edfbc487e468d6219ebeebf6e36702e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/997830a36967c65110d033f84f7ba67a8edfbc487e468d6219ebeebf6e36702e/userdata/shm major:0 minor:357 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ea9fdda0d4f51060b0fd95750c35c3a585170ff17cd0767f6e759856bf67cb8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ea9fdda0d4f51060b0fd95750c35c3a585170ff17cd0767f6e759856bf67cb8/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f18bbf3213ffb7fe24c465bdcea01d198c6b008dadf0db37a1dfc112c2a5a3a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f18bbf3213ffb7fe24c465bdcea01d198c6b008dadf0db37a1dfc112c2a5a3a/userdata/shm major:0 minor:282 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a31b7e522fd5db8b986c7fc5a759eee40e3a666bdc4e0c2c07f4fcdfc78868d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a31b7e522fd5db8b986c7fc5a759eee40e3a666bdc4e0c2c07f4fcdfc78868d8/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab39d17504c1ed393c39b9ff8d73ce1229730f78105dc51229457d973aaf7a67/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab39d17504c1ed393c39b9ff8d73ce1229730f78105dc51229457d973aaf7a67/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4bd5e56fe730a54bdb7262f4cc4570f330c25c484754253c6f2e2786da61949/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4bd5e56fe730a54bdb7262f4cc4570f330c25c484754253c6f2e2786da61949/userdata/shm major:0 minor:366 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e55134de16b156aa4d627cbd1bab62c88209475371be2f582b51a1c9c0e6ffe3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e55134de16b156aa4d627cbd1bab62c88209475371be2f582b51a1c9c0e6ffe3/userdata/shm major:0 minor:358 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eee132e0769005a7f31e3d3b4c1f929f8d96f2bc565566ef9b26c3b9bfaf298a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eee132e0769005a7f31e3d3b4c1f929f8d96f2bc565566ef9b26c3b9bfaf298a/userdata/shm major:0 minor:345 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f149510e7151cb1a40537997bd10ed25b77594f9c712d5dc4faf1502efbd1b71/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f149510e7151cb1a40537997bd10ed25b77594f9c712d5dc4faf1502efbd1b71/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ffdadb7c0f31868d1a63d3ad83d08d3bbe287f99b99b656a23ba9a7f3d00cb81/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ffdadb7c0f31868d1a63d3ad83d08d3bbe287f99b99b656a23ba9a7f3d00cb81/userdata/shm major:0 minor:314 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/062e4d3d-4460-4412-93d3-22681a9a74e6/volumes/kubernetes.io~projected/kube-api-access-kgbnp:{mountpoint:/var/lib/kubelet/pods/062e4d3d-4460-4412-93d3-22681a9a74e6/volumes/kubernetes.io~projected/kube-api-access-kgbnp major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/062e4d3d-4460-4412-93d3-22681a9a74e6/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/062e4d3d-4460-4412-93d3-22681a9a74e6/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08da34c7-c69a-4eb9-848a-399cab14840e/volumes/kubernetes.io~projected/kube-api-access-2w596:{mountpoint:/var/lib/kubelet/pods/08da34c7-c69a-4eb9-848a-399cab14840e/volumes/kubernetes.io~projected/kube-api-access-2w596 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08da34c7-c69a-4eb9-848a-399cab14840e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/08da34c7-c69a-4eb9-848a-399cab14840e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16b948f8-e3da-4285-8342-27cbd2dbd141/volumes/kubernetes.io~projected/kube-api-access-pwzkk:{mountpoint:/var/lib/kubelet/pods/16b948f8-e3da-4285-8342-27cbd2dbd141/volumes/kubernetes.io~projected/kube-api-access-pwzkk major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/400b676a-a3a0-459d-9da6-8d30b2d3b497/volumes/kubernetes.io~projected/kube-api-access-n4ttf:{mountpoint:/var/lib/kubelet/pods/400b676a-a3a0-459d-9da6-8d30b2d3b497/volumes/kubernetes.io~projected/kube-api-access-n4ttf major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50818995-4b65-457e-9b55-250b548e80d3/volumes/kubernetes.io~projected/kube-api-access-b8ncx:{mountpoint:/var/lib/kubelet/pods/50818995-4b65-457e-9b55-250b548e80d3/volumes/kubernetes.io~projected/kube-api-access-b8ncx major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50818995-4b65-457e-9b55-250b548e80d3/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/50818995-4b65-457e-9b55-250b548e80d3/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55f1a18a-59ef-4146-a0e8-3fc632ab20f5/volumes/kubernetes.io~projected/kube-api-access-z82fk:{mountpoint:/var/lib/kubelet/pods/55f1a18a-59ef-4146-a0e8-3fc632ab20f5/volumes/kubernetes.io~projected/kube-api-access-z82fk major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/571b679b-fad2-423d-96e5-8d5550bb536d/volumes/kubernetes.io~projected/kube-api-access-r8b2g:{mountpoint:/var/lib/kubelet/pods/571b679b-fad2-423d-96e5-8d5550bb536d/volumes/kubernetes.io~projected/kube-api-access-r8b2g major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/575aa82a-6d21-46c1-80ff-7f756f00c108/volumes/kubernetes.io~projected/kube-api-access-pg9j9:{mountpoint:/var/lib/kubelet/pods/575aa82a-6d21-46c1-80ff-7f756f00c108/volumes/kubernetes.io~projected/kube-api-access-pg9j9 major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/575aa82a-6d21-46c1-80ff-7f756f00c108/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/575aa82a-6d21-46c1-80ff-7f756f00c108/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/575aa82a-6d21-46c1-80ff-7f756f00c108/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/575aa82a-6d21-46c1-80ff-7f756f00c108/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volume-subpaths/web-config/config-reloader/2 major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volume-subpaths/web-config/prometheus/15 major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~empty-dir/config-out major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~projected/kube-api-access-6k6ph:{mountpoint:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~projected/kube-api-access-6k6ph major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~projected/tls-assets major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/config major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/web-config major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c/volumes/kubernetes.io~projected/kube-api-access-dkr86:{mountpoint:/var/lib/kubelet/pods/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c/volumes/kubernetes.io~projected/kube-api-access-dkr86 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d877fe5-7b94-4645-bff5-6df0796bb637/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/7d877fe5-7b94-4645-bff5-6df0796bb637/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bae2714-9c1b-44db-a6ef-b0b008cf1fa2/volumes/kubernetes.io~projected/kube-api-access-jq8hk:{mountpoint:/var/lib/kubelet/pods/8bae2714-9c1b-44db-a6ef-b0b008cf1fa2/volumes/kubernetes.io~projected/kube-api-access-jq8hk major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bae2714-9c1b-44db-a6ef-b0b008cf1fa2/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/8bae2714-9c1b-44db-a6ef-b0b008cf1fa2/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f247bb3-6322-43c8-bfad-f27a41e26500/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/8f247bb3-6322-43c8-bfad-f27a41e26500/volumes/kubernetes.io~secret/agent-certs major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/904f42d7-69e3-462f-9055-b6b840800f13/volumes/kubernetes.io~projected/kube-api-access-7hnbq:{mountpoint:/var/lib/kubelet/pods/904f42d7-69e3-462f-9055-b6b840800f13/volumes/kubernetes.io~projected/kube-api-access-7hnbq major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92cf7508-2fc5-49a5-9e71-5d9c73bf200e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/92cf7508-2fc5-49a5-9e71-5d9c73bf200e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92cf7508-2fc5-49a5-9e71-5d9c73bf200e/volumes/kubernetes.io~projected/kube-api-access-pqks5:{mountpoint:/var/lib/kubelet/pods/92cf7508-2fc5-49a5-9e71-5d9c73bf200e/volumes/kubernetes.io~projected/kube-api-access-pqks5 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92cf7508-2fc5-49a5-9e71-5d9c73bf200e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/92cf7508-2fc5-49a5-9e71-5d9c73bf200e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf66bb31-fee6-49fd-aaea-c8e8d8361c77/volumes/kubernetes.io~projected/kube-api-access-q8bkq:{mountpoint:/var/lib/kubelet/pods/cf66bb31-fee6-49fd-aaea-c8e8d8361c77/volumes/kubernetes.io~projected/kube-api-access-q8bkq major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf66bb31-fee6-49fd-aaea-c8e8d8361c77/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/cf66bb31-fee6-49fd-aaea-c8e8d8361c77/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf66bb31-fee6-49fd-aaea-c8e8d8361c77/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/cf66bb31-fee6-49fd-aaea-c8e8d8361c77/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~projected/kube-api-access-vkbnh:{mountpoint:/var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~projected/kube-api-access-vkbnh major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d680db8a-15e4-4817-bff1-a420f9f54681/volumes/kubernetes.io~projected/kube-api-access-6w65q:{mountpoint:/var/lib/kubelet/pods/d680db8a-15e4-4817-bff1-a420f9f54681/volumes/kubernetes.io~projected/kube-api-access-6w65q major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d680db8a-15e4-4817-bff1-a420f9f54681/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/d680db8a-15e4-4817-bff1-a420f9f54681/volumes/kubernetes.io~secret/metrics-certs major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7a31b67-a1b9-4b6a-a112-69cd5c833019/volumes/kubernetes.io~projected/kube-api-access-2qsw4:{mountpoint:/var/lib/kubelet/pods/d7a31b67-a1b9-4b6a-a112-69cd5c833019/volumes/kubernetes.io~projected/kube-api-access-2qsw4 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d92bdf4f-f9e4-424c-9c87-ec015574ff1a/volumes/kubernetes.io~projected/kube-api-access-7c47g:{mountpoint:/var/lib/kubelet/pods/d92bdf4f-f9e4-424c-9c87-ec015574ff1a/volumes/kubernetes.io~projected/kube-api-access-7c47g major:0 minor:361 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4046f43ddbd564275e31103e9f07b1b1a8897d11f2a6c18f34cf7bd0ae9ecafa/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/80ea7d5cc4ebc969156c2550b98c3149fcea5fcb9e0fb0cee2dfcc914e3534a2/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f2bb94e6e397457aa344409b970576ad66aa0bd926aa56af071aa2677da06b80/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/2ab69660cf08fa7f63a71a0b1ba90adb46948d531213a783d74b10c558b8b21c/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/9cce1f7304484864923e6b2960167f44e055f351a8fc265c2e570c0152cbee65/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/912f033222c994fe7ce771ab40bdc11ef18069c7e88f154b1f3b39aafb3d217c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/86103ced350fee01fa05300c84fd1e07356b49183f7edde57004f1b77c1eb01f/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/bdc10aa02e0596518b6519030f65418c76fe7b720129bde557f773cf89b2110a/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/ca5d3ba55b30e9abd4dc8544bbd2089e9a5badf1cb099709e9a9d966079df15f/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/f7cc4cad3eeaca4e751a503b5db2741e0df174437a718c4bda77ced2afb78182/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/f8349c3d17b68d170a51a03145d51b8439276fefa560868232ea68dcbcbb1496/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/b03f9ada59adddd8b5d3f95b3ae48aa6cccee6d00f6ceeb18b4a3cdf806d4c7b/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/7220785ff241783a5e3510c9a267df77fe3a4ffcc322086025fec5bcbecca06b/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/0cc39e52e51ab73228bbe4fe4c026998b482a0e8a392cd0a3a0453d2bc893006/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/20d211dc6c712f0952be4d0662405bd52db10b9d51670c2a0ff3993448b61eff/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/4885800cc02166cfa6060a85b93b6689cdd502c5662a929ad666901a10c24c71/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-168:{mountpoint:/var/lib/containers/storage/overlay/13025ab3374a87c310ad270b3e805a1493ed8b04c50c8ebaf4da1e34a75d3fbf/merged major:0 minor:168 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/4bb713b78934c99bbd28616c3a337ef6b86c60f45a3dd1775ef7a6a54736909b/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/d4ffd83f6c6e1752c8bbee86505aaa558ab3bbd206ee8db1cdf3767fc9036320/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/2d798a8b53f28d62c1dba5ce91e22742c5df7167d6c4a380d97cf4d8ece2ef48/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c95019c90dfe33a9c3ac14c36bd5b1465900e07fcc31b14162953f695c727bb1/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/2d76ae662d050dce7f5728bccb51af6d9d6c72ef228cb2f05996a521109558f1/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/8f65339d73e57f1f258b2b0c1b59925cb636ac9fd61dbbf1810d954a961ccfbd/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/d8f086d2139a3891cbba56d1603c0810b0c6e5663a85eba261af72a01c97649e/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/7fe9e6ce2f875216343a518ef47f07494e0b5877a8cb8e408334bd3f0ec4ec48/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/766845a055dd70a0d183a1bb82c3066b762d9130cb68c1f36a60372d4c8f20fb/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/84ee64f6ecba3197d19ca26c984050d2882d7089dda1720ca15e458477b799dc/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/41d85652ed85264c3295cf59ddef82c9a31bd00d340fc45bf69388722338c62d/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/d4e4dd6027a10d7aec5af3a2f1723b67b873a049f19d863a0d0a3cf855175721/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/ae391ad936692080268d6af2ce5ce50443a7871160f8b7c81e6d9a54a1e54b4a/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/0c0e2f2b7059f17901d0f869aa17dd12cb0dfbda050533f0ec365a9a6fae122d/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/41b31aa1ff11b186487da0bace0fa061bcfacf4d010fbcf1312fa1bae56dc09d/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/371a36f63b016c14a7c2f26c4bc48d0d42e9ce17fe27d5f30794ab5a19745dae/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/77246dbf0430366306b09e4a6b47421a8602592d125b973ac00e02b4216dcd09/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/f9a3821d5399780420df3d78892361cea3ba30fa4e33a59f570ed7a686e8fb0b/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/cbb10c7251409eee4a19a0ecefb1a47190a9804d9c78d0327802d04c8c3b1261/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/bd934b09c6c317d50255e8fa7a6dbb42ed8721260b5e9eafc7560931319cf7a2/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/d774f9c193fa5423dab0698dfa40f9c8a52280a45a012adc0f3c8b2e3119a7cb/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/5f5309fb0fc5defc070e8b19e9fd2cf2db7a5d23e87af145070196bd25c9024e/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/f9220bd666687ab2e24d8eb33409b53b70b4654c36e969aab3ddb99e3e3ed371/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/38ce4c6777f145ab7399dd6002fd7448cccab9c3027f05817cf6349586c43d50/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/fa72440efa1c12e72c07bd93c9190defbe2a7f58e8edd5ed7afed4227335f4c0/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/ce1fffd958c87bc28e3747460c780cc088e0059c7dcf38829b819eb1f3b3d4fb/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/401010e57428e36e5c7d63d7650318752412f4168dfc8e13fa5a13baf95fd800/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/c7c19d94302b906d82784ecc0108bfae9e27cc21893c27dc7a8929c3d43aa26b/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/7af2e49ee91cea0660097edd9cb8d666e46dc5e727b28aa44012fd4e435e10db/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/a342920f5d50b2f3bbdc71bfa4b1da5605580b44c1506df1bb3be28b0af63902/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/a806db4a7480ba482619ace95bf63481789b5385a232ba349e267139efd9c25f/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/0e24d8fd1d2aa5f04638488c56d1f2be91c079676584bb0163c3af4864d321ed/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/9227d75ef761f8341110d06d5ef1862c43ad113a7ef7e29fdf95e18afac2f2fa/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/b201b85bc1b6766fef6a53cc31de65557c209a0fde40d96a223cb8f093c54752/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/ce1daacf2a72ac95b1bcf825c638cdc1c683f1d922521909b876b292d18dcad6/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/16b6f606510cae4260b6e11f29c42eeed6a23320a3fdab7da1eeca3c0c33cc17/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/13b0d451ebd9d122540d20179b124da661e0b96605b6a6e90e56a08092a26eb5/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/fdc08732acf5c367cb182796d63d80548d2ef233f9a345b3bca037bfc148ae0c/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/6715b6c8961630417e435d41b0d73b35e3ea24625998d7e51253f00b04cdb7bc/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/97c4927847428ac2d46707a4e47dc700b54b7a466ecb9ba2a589c083d2cc64cb/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/43cb92d8acf491802b08df51f77fbca321aa853c680b42d740223dff4f03cc4d/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/2b25b385e09e7d7185c8b4c1c7fc0b1b6f66e9bcf47e9197c83c8aff7165424a/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/ac9f761b248ad37ffe9a5491e2ded7e8423104294b6d510456fe6e22e99ff9f4/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/18812db555995c0881aca03258a74f341a500f066d37eb4d2cf441d274376f79/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/0fd42f26cd3d4a0f0d5f95626561200bb352dc37fc1b072a6f924a7371428c0b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/133d74429b5ac071102bdf3f5ca952a39a3a06c070bd0ee00fb731606fab984e/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/89be4864200a695869747020475a4e7db954c6c5b9453a1688efc585e180b1da/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/a87a6b7d23c9256b013907e23a11ec6040305e0327057832b1c1fc2005b0dc06/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/36b074204d6ccbf7f3c1f2008d06919f2dce5e53576fc6830ecadfb87c8c7369/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/1628a9ad42d6a909300cc2b920d96439b0e343ab2a7805a60fd331600f2b353a/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/09bdd51f64a529fb9f920db44856db7c5df805df7c3ea256f2d1409ff71cb280/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/4d471015976bcdbf40a524ee3937cac6973f410e0cd6676ec911929fe86ab450/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 03:16:05.525059 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.524608 5490 manager.go:217] Machine: {Timestamp:2026-10-31 03:16:05.524128452 +0000 UTC m=+0.058823397 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101872 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22067afb54a9236ebcbc456fcdeb33 SystemUUID:ec22067a-fb54-a923-6ebc-bc456fcdeb33 BootID:4a3470f5-09fb-46a8-a9b3-843ec9a4c60b Filesystems:[{Device:/run/containers/storage/overlay-containers/997830a36967c65110d033f84f7ba67a8edfbc487e468d6219ebeebf6e36702e/userdata/shm DeviceMajor:0 DeviceMinor:357 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d92bdf4f-f9e4-424c-9c87-ec015574ff1a/volumes/kubernetes.io~projected/kube-api-access-7c47g DeviceMajor:0 DeviceMinor:361 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/92cf7508-2fc5-49a5-9e71-5d9c73bf200e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/575aa82a-6d21-46c1-80ff-7f756f00c108/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:265 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:277 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f149510e7151cb1a40537997bd10ed25b77594f9c712d5dc4faf1502efbd1b71/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/55f1a18a-59ef-4146-a0e8-3fc632ab20f5/volumes/kubernetes.io~projected/kube-api-access-z82fk DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/062e4d3d-4460-4412-93d3-22681a9a74e6/volumes/kubernetes.io~projected/kube-api-access-kgbnp DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:278 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:280 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf66bb31-fee6-49fd-aaea-c8e8d8361c77/volumes/kubernetes.io~projected/kube-api-access-q8bkq DeviceMajor:0 DeviceMinor:229 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:336 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e55134de16b156aa4d627cbd1bab62c88209475371be2f582b51a1c9c0e6ffe3/userdata/shm DeviceMajor:0 DeviceMinor:358 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-168 DeviceMajor:0 DeviceMinor:168 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:334 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/400b676a-a3a0-459d-9da6-8d30b2d3b497/volumes/kubernetes.io~projected/kube-api-access-n4ttf DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~projected/kube-api-access-vkbnh DeviceMajor:0 DeviceMinor:281 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/08da34c7-c69a-4eb9-848a-399cab14840e/volumes/kubernetes.io~projected/kube-api-access-2w596 DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0618767972e39b44ad2edae67be30a4b1b9c31bbe1a2be109a292f34c78d17e1/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f18bbf3213ffb7fe24c465bdcea01d198c6b008dadf0db37a1dfc112c2a5a3a/userdata/shm DeviceMajor:0 DeviceMinor:282 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d680db8a-15e4-4817-bff1-a420f9f54681/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:355 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33995e191d31be212d6ba7963974885157df6f40f818511415175c4c39f249c1/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/383c4ca43cffb50de3458004d549df509bcc5995b59f83b821a5e9036ea423eb/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08da34c7-c69a-4eb9-848a-399cab14840e/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:279 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/571b679b-fad2-423d-96e5-8d5550bb536d/volumes/kubernetes.io~projected/kube-api-access-r8b2g DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/16b948f8-e3da-4285-8342-27cbd2dbd141/volumes/kubernetes.io~projected/kube-api-access-pwzkk DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/945bab7a9e73e6bbd58ee18321120992a5f33158baf662d521741b3343e8b265/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/062e4d3d-4460-4412-93d3-22681a9a74e6/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cf66bb31-fee6-49fd-aaea-c8e8d8361c77/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:230 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/575aa82a-6d21-46c1-80ff-7f756f00c108/volumes/kubernetes.io~projected/kube-api-access-pg9j9 DeviceMajor:0 DeviceMinor:270 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:276 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8bae2714-9c1b-44db-a6ef-b0b008cf1fa2/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:309 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:338 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d680db8a-15e4-4817-bff1-a420f9f54681/volumes/kubernetes.io~projected/kube-api-access-6w65q DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d395aca5-b8ff-4da1-ad31-dd26e2427c2e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:275 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eee132e0769005a7f31e3d3b4c1f929f8d96f2bc565566ef9b26c3b9bfaf298a/userdata/shm DeviceMajor:0 DeviceMinor:345 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92cf7508-2fc5-49a5-9e71-5d9c73bf200e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:340 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c/volumes/kubernetes.io~projected/kube-api-access-dkr86 DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ffdadb7c0f31868d1a63d3ad83d08d3bbe287f99b99b656a23ba9a7f3d00cb81/userdata/shm DeviceMajor:0 DeviceMinor:314 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:341 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:338 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/904f42d7-69e3-462f-9055-b6b840800f13/volumes/kubernetes.io~projected/kube-api-access-7hnbq DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/09895ee45100c465e50e818c21642f320e820aa5cc3885d4a329c06f9496167c/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84059b6591690137309e98aa0e50941340cb9d7a1ce880e7e93223834fdb32e4/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:330 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:338 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:343 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cf66bb31-fee6-49fd-aaea-c8e8d8361c77/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:339 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~projected/kube-api-access-6k6ph DeviceMajor:0 DeviceMinor:342 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c37acdd97b379138de5515cee74a2e79bb0dcb22b5a4ce180156ce646d44942/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9464aa9bf3b2aa43daa2a9323f4fe8a5b4c275c88cedd1e89755a98a3df10be7/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:338 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab39d17504c1ed393c39b9ff8d73ce1229730f78105dc51229457d973aaf7a67/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3829dc4f52ebb213c7ce11bfd34086fe601be80ee39a8c9a63e8b1a4c079c258/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bae2714-9c1b-44db-a6ef-b0b008cf1fa2/volumes/kubernetes.io~projected/kube-api-access-jq8hk DeviceMajor:0 DeviceMinor:313 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:335 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/575aa82a-6d21-46c1-80ff-7f756f00c108/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50818995-4b65-457e-9b55-250b548e80d3/volumes/kubernetes.io~projected/kube-api-access-b8ncx DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:344 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d877fe5-7b94-4645-bff5-6df0796bb637/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:356 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4bd5e56fe730a54bdb7262f4cc4570f330c25c484754253c6f2e2786da61949/userdata/shm DeviceMajor:0 DeviceMinor:366 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/92cf7508-2fc5-49a5-9e71-5d9c73bf200e/volumes/kubernetes.io~projected/kube-api-access-pqks5 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f247bb3-6322-43c8-bfad-f27a41e26500/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ea9fdda0d4f51060b0fd95750c35c3a585170ff17cd0767f6e759856bf67cb8/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a31b7e522fd5db8b986c7fc5a759eee40e3a666bdc4e0c2c07f4fcdfc78868d8/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31ec223e35e7af6752b1dbb7ca38857f538db7258f801f5630640b324945bb38/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:337 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d7a31b67-a1b9-4b6a-a112-69cd5c833019/volumes/kubernetes.io~projected/kube-api-access-2qsw4 DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01fb80cbee538275da581600b645f0018e78e8a5438ae2314c668976efc95d2f/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50818995-4b65-457e-9b55-250b548e80d3/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0618767972e39b4 MacAddress:16:5a:98:13:87:2d Speed:10000 Mtu:8901} {Name:33995e191d31be2 MacAddress:0e:5b:f6:f9:c4:f6 Speed:10000 Mtu:8901} {Name:383c4ca43cffb50 MacAddress:7a:41:af:fd:56:9d Speed:10000 Mtu:8901} {Name:997830a36967c65 MacAddress:ca:a5:f1:38:58:1d Speed:10000 Mtu:8901} {Name:9f18bbf3213ffb7 MacAddress:aa:93:cb:50:5c:84 Speed:10000 Mtu:8901} {Name:a31b7e522fd5db8 MacAddress:0a:fb:5b:80:da:48 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cc:5f:23:55 Speed:0 Mtu:9001} {Name:br-int MacAddress:ae:df:d7:08:bc:c1 Speed:0 Mtu:8901} {Name:c4bd5e56fe730a5 MacAddress:6e:60:ad:44:5b:03 Speed:10000 Mtu:8901} {Name:e55134de16b156a MacAddress:ae:d2:24:90:ca:3f Speed:10000 Mtu:8901} {Name:eee132e0769005a MacAddress:8e:0e:6f:29:5b:ef Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:cc:5f:23:55 Speed:0 Mtu:9001} {Name:ffdadb7c0f31868 MacAddress:26:55:91:bf:a4:e1 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:d6:7e:a9:34:81:54 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4e:c1:c5:09:a2:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 03:16:05.525059 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.524990 5490 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 03:16:05.525059 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.525036 5490 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 03:16:05.525328 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.525300 5490 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 03:16:05.525455 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.525327 5490 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-76.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 03:16:05.525495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.525457 5490 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 03:16:05.525495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.525465 5490 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 03:16:05.525495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.525475 5490 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 03:16:05.525572 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.525495 5490 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 03:16:05.525606 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.525592 5490 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 03:16:05.525778 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.525760 5490 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 03:16:05.525888 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.525880 5490 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 03:16:05.525924 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.525916 5490 kubelet.go:493] "Attempting to sync node with API server" Oct 31 03:16:05.525951 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.525930 5490 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 03:16:05.525951 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.525940 5490 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 03:16:05.525951 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.525948 5490 kubelet.go:405] "Adding apiserver pod source" Oct 31 03:16:05.526031 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.525958 5490 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 03:16:05.526580 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.526567 5490 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 03:16:05.526580 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.526580 5490 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 03:16:05.527085 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.527064 5490 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 03:16:05.530568 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.530554 5490 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 03:16:05.530616 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.530569 5490 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 03:16:05.530702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.530690 5490 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 03:16:05.530894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.530882 5490 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 03:16:05.530924 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.530912 5490 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 03:16:05.531028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.531018 5490 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 03:16:05.531061 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.531033 5490 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 03:16:05.531061 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.531039 5490 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 03:16:05.531061 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.531046 5490 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 03:16:05.531061 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.531055 5490 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 03:16:05.531177 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.531066 5490 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 03:16:05.531177 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.531074 5490 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 03:16:05.531177 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.531079 5490 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 03:16:05.531177 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.531086 5490 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 03:16:05.531177 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.531092 5490 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 03:16:05.531177 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.531101 5490 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 03:16:05.531177 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.531111 5490 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 03:16:05.531177 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.531142 5490 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 03:16:05.531177 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.531151 5490 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 03:16:05.534721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.534704 5490 server.go:1263] "Started kubelet" Oct 31 03:16:05.535859 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.534776 5490 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 03:16:05.535859 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.534969 5490 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 03:16:05.535859 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.535291 5490 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 03:16:05.535859 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.535656 5490 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 03:16:05.535348 ip-10-0-142-76 systemd[1]: Started Kubernetes Kubelet. Oct 31 03:16:05.536443 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.536426 5490 server.go:317] "Adding debug handlers to kubelet server" Oct 31 03:16:05.544049 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.542391 5490 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 03:16:05.544049 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.542491 5490 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 03:16:05.544049 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:05.542547 5490 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-76.us-east-2.compute.internal\" not found" Oct 31 03:16:05.544049 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.542557 5490 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 03:16:05.544049 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.542575 5490 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 03:16:05.544049 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.542690 5490 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 03:16:05.544049 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.543023 5490 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:09:56 +0000 UTC" deadline="2028-05-30 07:10:34.601588195 +0000 UTC" Oct 31 03:16:05.544049 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.543059 5490 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13851h54m29.058532693s" Oct 31 03:16:05.545180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545145 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="904f42d7-69e3-462f-9055-b6b840800f13" volumeName="kubernetes.io/configmap/904f42d7-69e3-462f-9055-b6b840800f13-multus-daemon-config" seLinuxMountContext="" Oct 31 03:16:05.545220 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545185 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" volumeName="kubernetes.io/projected/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-kube-api-access-vkbnh" seLinuxMountContext="" Oct 31 03:16:05.545220 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545198 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f247bb3-6322-43c8-bfad-f27a41e26500" volumeName="kubernetes.io/secret/8f247bb3-6322-43c8-bfad-f27a41e26500-agent-certs" seLinuxMountContext="" Oct 31 03:16:05.545220 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545212 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50818995-4b65-457e-9b55-250b548e80d3" volumeName="kubernetes.io/secret/50818995-4b65-457e-9b55-250b548e80d3-cert" seLinuxMountContext="" Oct 31 03:16:05.545296 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545225 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" volumeName="kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 31 03:16:05.545296 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545237 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" volumeName="kubernetes.io/empty-dir/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-config-out" seLinuxMountContext="" Oct 31 03:16:05.545296 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545248 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" volumeName="kubernetes.io/projected/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-tls-assets" seLinuxMountContext="" Oct 31 03:16:05.545296 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545261 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ad1f3a7-eea4-4104-b3ff-e860da35bf9c" volumeName="kubernetes.io/empty-dir/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-tmp" seLinuxMountContext="" Oct 31 03:16:05.545296 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545275 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bae2714-9c1b-44db-a6ef-b0b008cf1fa2" volumeName="kubernetes.io/projected/8bae2714-9c1b-44db-a6ef-b0b008cf1fa2-kube-api-access-jq8hk" seLinuxMountContext="" Oct 31 03:16:05.545296 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545288 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" volumeName="kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 31 03:16:05.545457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545300 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="062e4d3d-4460-4412-93d3-22681a9a74e6" volumeName="kubernetes.io/projected/062e4d3d-4460-4412-93d3-22681a9a74e6-kube-api-access-kgbnp" seLinuxMountContext="" Oct 31 03:16:05.545457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545312 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="571b679b-fad2-423d-96e5-8d5550bb536d" volumeName="kubernetes.io/projected/571b679b-fad2-423d-96e5-8d5550bb536d-kube-api-access-r8b2g" seLinuxMountContext="" Oct 31 03:16:05.545457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545323 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" volumeName="kubernetes.io/empty-dir/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-db" seLinuxMountContext="" Oct 31 03:16:05.545457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545334 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" volumeName="kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 31 03:16:05.545457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545347 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" volumeName="kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 31 03:16:05.545457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545367 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92cf7508-2fc5-49a5-9e71-5d9c73bf200e" volumeName="kubernetes.io/projected/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-kube-api-access-pqks5" seLinuxMountContext="" Oct 31 03:16:05.545457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545379 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92cf7508-2fc5-49a5-9e71-5d9c73bf200e" volumeName="kubernetes.io/secret/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 03:16:05.545457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545391 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf66bb31-fee6-49fd-aaea-c8e8d8361c77" volumeName="kubernetes.io/secret/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-prometheus-operator-tls" seLinuxMountContext="" Oct 31 03:16:05.545457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545404 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f247bb3-6322-43c8-bfad-f27a41e26500" volumeName="kubernetes.io/configmap/8f247bb3-6322-43c8-bfad-f27a41e26500-konnectivity-ca" seLinuxMountContext="" Oct 31 03:16:05.545457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545454 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" volumeName="kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 03:16:05.545738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545467 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7a31b67-a1b9-4b6a-a112-69cd5c833019" volumeName="kubernetes.io/configmap/d7a31b67-a1b9-4b6a-a112-69cd5c833019-cni-binary-copy" seLinuxMountContext="" Oct 31 03:16:05.545738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545480 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7a31b67-a1b9-4b6a-a112-69cd5c833019" volumeName="kubernetes.io/projected/d7a31b67-a1b9-4b6a-a112-69cd5c833019-kube-api-access-2qsw4" seLinuxMountContext="" Oct 31 03:16:05.545738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545503 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="062e4d3d-4460-4412-93d3-22681a9a74e6" volumeName="kubernetes.io/secret/062e4d3d-4460-4412-93d3-22681a9a74e6-metrics-tls" seLinuxMountContext="" Oct 31 03:16:05.545738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545516 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08da34c7-c69a-4eb9-848a-399cab14840e" volumeName="kubernetes.io/empty-dir/08da34c7-c69a-4eb9-848a-399cab14840e-data-volume" seLinuxMountContext="" Oct 31 03:16:05.545738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545529 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="575aa82a-6d21-46c1-80ff-7f756f00c108" volumeName="kubernetes.io/empty-dir/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-textfile" seLinuxMountContext="" Oct 31 03:16:05.545738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545579 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="904f42d7-69e3-462f-9055-b6b840800f13" volumeName="kubernetes.io/projected/904f42d7-69e3-462f-9055-b6b840800f13-kube-api-access-7hnbq" seLinuxMountContext="" Oct 31 03:16:05.545738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545593 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92cf7508-2fc5-49a5-9e71-5d9c73bf200e" volumeName="kubernetes.io/configmap/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-env-overrides" seLinuxMountContext="" Oct 31 03:16:05.545738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545606 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf66bb31-fee6-49fd-aaea-c8e8d8361c77" volumeName="kubernetes.io/secret/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 03:16:05.545738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545628 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" volumeName="kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 31 03:16:05.545738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545647 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" volumeName="kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 31 03:16:05.545738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545661 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf66bb31-fee6-49fd-aaea-c8e8d8361c77" volumeName="kubernetes.io/configmap/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-metrics-client-ca" seLinuxMountContext="" Oct 31 03:16:05.545738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545712 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" volumeName="kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-tls" seLinuxMountContext="" Oct 31 03:16:05.545738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545726 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16b948f8-e3da-4285-8342-27cbd2dbd141" volumeName="kubernetes.io/projected/16b948f8-e3da-4285-8342-27cbd2dbd141-kube-api-access-pwzkk" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545742 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55f1a18a-59ef-4146-a0e8-3fc632ab20f5" volumeName="kubernetes.io/configmap/55f1a18a-59ef-4146-a0e8-3fc632ab20f5-serviceca" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545769 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="575aa82a-6d21-46c1-80ff-7f756f00c108" volumeName="kubernetes.io/configmap/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545782 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" volumeName="kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545793 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" volumeName="kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545804 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" volumeName="kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-metrics-client-certs" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545816 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7a31b67-a1b9-4b6a-a112-69cd5c833019" volumeName="kubernetes.io/configmap/d7a31b67-a1b9-4b6a-a112-69cd5c833019-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545828 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" volumeName="kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545839 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16b948f8-e3da-4285-8342-27cbd2dbd141" volumeName="kubernetes.io/empty-dir/16b948f8-e3da-4285-8342-27cbd2dbd141-tmp-dir" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545852 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="575aa82a-6d21-46c1-80ff-7f756f00c108" volumeName="kubernetes.io/secret/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-tls" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545865 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="575aa82a-6d21-46c1-80ff-7f756f00c108" volumeName="kubernetes.io/secret/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545915 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ad1f3a7-eea4-4104-b3ff-e860da35bf9c" volumeName="kubernetes.io/projected/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-kube-api-access-dkr86" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545930 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08da34c7-c69a-4eb9-848a-399cab14840e" volumeName="kubernetes.io/secret/08da34c7-c69a-4eb9-848a-399cab14840e-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545946 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="400b676a-a3a0-459d-9da6-8d30b2d3b497" volumeName="kubernetes.io/projected/400b676a-a3a0-459d-9da6-8d30b2d3b497-kube-api-access-n4ttf" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545962 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="575aa82a-6d21-46c1-80ff-7f756f00c108" volumeName="kubernetes.io/projected/575aa82a-6d21-46c1-80ff-7f756f00c108-kube-api-access-pg9j9" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545976 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92cf7508-2fc5-49a5-9e71-5d9c73bf200e" volumeName="kubernetes.io/configmap/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-ovnkube-config" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.545990 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" volumeName="kubernetes.io/configmap/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-metrics-client-ca" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546031 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" volumeName="kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546048 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="062e4d3d-4460-4412-93d3-22681a9a74e6" volumeName="kubernetes.io/empty-dir/062e4d3d-4460-4412-93d3-22681a9a74e6-tmp-dir" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546063 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08da34c7-c69a-4eb9-848a-399cab14840e" volumeName="kubernetes.io/projected/08da34c7-c69a-4eb9-848a-399cab14840e-kube-api-access-2w596" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546077 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d877fe5-7b94-4645-bff5-6df0796bb637" volumeName="kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546091 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92cf7508-2fc5-49a5-9e71-5d9c73bf200e" volumeName="kubernetes.io/configmap/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-ovnkube-script-lib" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546102 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d680db8a-15e4-4817-bff1-a420f9f54681" volumeName="kubernetes.io/projected/d680db8a-15e4-4817-bff1-a420f9f54681-kube-api-access-6w65q" seLinuxMountContext="" Oct 31 03:16:05.546115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546114 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7a31b67-a1b9-4b6a-a112-69cd5c833019" volumeName="kubernetes.io/configmap/d7a31b67-a1b9-4b6a-a112-69cd5c833019-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546146 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" volumeName="kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546161 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d680db8a-15e4-4817-bff1-a420f9f54681" volumeName="kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546172 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="062e4d3d-4460-4412-93d3-22681a9a74e6" volumeName="kubernetes.io/configmap/062e4d3d-4460-4412-93d3-22681a9a74e6-config-volume" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546184 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" volumeName="kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546197 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="904f42d7-69e3-462f-9055-b6b840800f13" volumeName="kubernetes.io/configmap/904f42d7-69e3-462f-9055-b6b840800f13-cni-binary-copy" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546208 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf66bb31-fee6-49fd-aaea-c8e8d8361c77" volumeName="kubernetes.io/projected/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-kube-api-access-q8bkq" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546220 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55f1a18a-59ef-4146-a0e8-3fc632ab20f5" volumeName="kubernetes.io/projected/55f1a18a-59ef-4146-a0e8-3fc632ab20f5-kube-api-access-z82fk" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546242 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="575aa82a-6d21-46c1-80ff-7f756f00c108" volumeName="kubernetes.io/configmap/575aa82a-6d21-46c1-80ff-7f756f00c108-metrics-client-ca" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546254 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" volumeName="kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546267 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" volumeName="kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-config" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546278 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ad1f3a7-eea4-4104-b3ff-e860da35bf9c" volumeName="kubernetes.io/empty-dir/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-tuned" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546289 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" volumeName="kubernetes.io/empty-dir/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-tmp" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546300 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" volumeName="kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-grpc-tls" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546313 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bae2714-9c1b-44db-a6ef-b0b008cf1fa2" volumeName="kubernetes.io/secret/8bae2714-9c1b-44db-a6ef-b0b008cf1fa2-monitoring-plugin-cert" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546324 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08da34c7-c69a-4eb9-848a-399cab14840e" volumeName="kubernetes.io/configmap/08da34c7-c69a-4eb9-848a-399cab14840e-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546336 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="571b679b-fad2-423d-96e5-8d5550bb536d" volumeName="kubernetes.io/configmap/571b679b-fad2-423d-96e5-8d5550bb536d-iptables-alerter-script" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546347 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" volumeName="kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-metrics-client-ca" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546358 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" volumeName="kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-web-config" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546369 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" volumeName="kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-grpc-tls" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546383 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50818995-4b65-457e-9b55-250b548e80d3" volumeName="kubernetes.io/projected/50818995-4b65-457e-9b55-250b548e80d3-kube-api-access-b8ncx" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546395 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" volumeName="kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546406 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" volumeName="kubernetes.io/projected/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-kube-api-access-6k6ph" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546418 5490 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" volumeName="kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546430 5490 reconstruct.go:97] "Volume reconstruction finished" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546438 5490 reconciler.go:29] "Reconciler: start to sync state" Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546863 5490 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546878 5490 factory.go:55] Registering systemd factory Oct 31 03:16:05.546922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.546887 5490 factory.go:223] Registration of the systemd container factory successfully Oct 31 03:16:05.548033 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.547932 5490 factory.go:153] Registering CRI-O factory Oct 31 03:16:05.548033 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.547949 5490 factory.go:223] Registration of the crio container factory successfully Oct 31 03:16:05.548033 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.547971 5490 factory.go:103] Registering Raw factory Oct 31 03:16:05.548033 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.547994 5490 manager.go:1199] Started watching for new ooms in manager Oct 31 03:16:05.548870 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.548395 5490 manager.go:319] Starting recovery of all containers Oct 31 03:16:05.550872 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.550855 5490 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:16:05.555599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.555572 5490 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:16:05.555971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.555946 5490 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:16:05.556293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.556246 5490 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 03:16:05.565639 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.565612 5490 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 03:16:05.567201 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.567179 5490 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 03:16:05.567300 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.567215 5490 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 03:16:05.567300 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.567235 5490 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 03:16:05.567300 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:05.567279 5490 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 03:16:05.575067 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.575044 5490 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 03:16:05.578446 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.578402 5490 generic.go:356] "Generic (PLEG): container finished" podID="9dcc8bbdd4365b42723f25cd49f22a1c" containerID="392fe58da98574e342eb491e1c359bbb7928bc4a5be06969278d16c476b7f327" exitCode=0 Oct 31 03:16:05.589372 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.589345 5490 generic.go:356] "Generic (PLEG): container finished" podID="575aa82a-6d21-46c1-80ff-7f756f00c108" containerID="b4959fbd23721abded557e3b3abdffee4d9625488608538b0b8730854c3b5e19" exitCode=0 Oct 31 03:16:05.602504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.602473 5490 generic.go:356] "Generic (PLEG): container finished" podID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerID="945c8971b228a65db2916c2b8a4ab87372c601d5d632272b26611e95c031c386" exitCode=0 Oct 31 03:16:05.610734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.610669 5490 generic.go:356] "Generic (PLEG): container finished" podID="d7a31b67-a1b9-4b6a-a112-69cd5c833019" containerID="5dd6cb9c9a3f579ab659b40bf4d18ff6d7151502802997687df56bde8260c025" exitCode=0 Oct 31 03:16:05.610850 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.610741 5490 generic.go:356] "Generic (PLEG): container finished" podID="d7a31b67-a1b9-4b6a-a112-69cd5c833019" containerID="a96dd3a582c40a0f95ea1ab2b4afdbcca86ab847366da1fe77a73f98a62d07a5" exitCode=0 Oct 31 03:16:05.610850 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.610765 5490 generic.go:356] "Generic (PLEG): container finished" podID="d7a31b67-a1b9-4b6a-a112-69cd5c833019" containerID="848c30e377a3fac13afe5f53b4fa4aee586581e0ef72e05f19ca121d7726dc0a" exitCode=0 Oct 31 03:16:05.610850 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.610787 5490 generic.go:356] "Generic (PLEG): container finished" podID="d7a31b67-a1b9-4b6a-a112-69cd5c833019" containerID="287e7c4db900ea9c7194c278321a5f96e1d6f2f49685cd9c4cb4a72e1f526ba6" exitCode=0 Oct 31 03:16:05.610850 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.610797 5490 generic.go:356] "Generic (PLEG): container finished" podID="d7a31b67-a1b9-4b6a-a112-69cd5c833019" containerID="045e623de2d0c1e257011bb7c3e358f459f5a0efb2533fad29fc08173ccedd17" exitCode=0 Oct 31 03:16:05.610850 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.610810 5490 generic.go:356] "Generic (PLEG): container finished" podID="d7a31b67-a1b9-4b6a-a112-69cd5c833019" containerID="712c2ede07b5be05256c54c6bdad4f5ff756eee935808477f12bfcf700177cc6" exitCode=0 Oct 31 03:16:05.649645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.649623 5490 manager.go:324] Recovery completed Oct 31 03:16:05.667738 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:05.667667 5490 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 03:16:05.675316 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.675287 5490 cpu_manager.go:226] "Starting" policy="none" Oct 31 03:16:05.675316 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.675310 5490 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 03:16:05.675508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.675333 5490 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 03:16:05.675508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.675497 5490 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 03:16:05.675588 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.675513 5490 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 03:16:05.675625 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.675609 5490 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 03:16:05.675625 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.675618 5490 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 03:16:05.675701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.675626 5490 policy_none.go:50] "Start" Oct 31 03:16:05.675701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.675637 5490 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 03:16:05.675701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.675650 5490 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 03:16:05.675866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.675851 5490 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 03:16:05.675896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.675868 5490 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 03:16:05.675896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.675876 5490 policy_none.go:44] "Start" Oct 31 03:16:05.681317 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.681280 5490 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 03:16:05.684031 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.684018 5490 manager.go:341] "Starting Device Plugin manager" Oct 31 03:16:05.684517 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:05.684044 5490 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 03:16:05.684570 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.684519 5490 server.go:87] "Starting device plugin registration server" Oct 31 03:16:05.684828 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.684815 5490 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 03:16:05.684885 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.684828 5490 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 03:16:05.685216 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.685195 5490 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 03:16:05.685377 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.685341 5490 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 03:16:05.685448 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.685387 5490 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 03:16:05.685448 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.685394 5490 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 03:16:05.685552 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.685452 5490 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:16:05.685373658Z","UUID":"b3b95c3e-b69e-42c7-9e56-30754a34a248","Handler":null,"Name":"","Endpoint":""} Oct 31 03:16:05.687630 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:05.686715 5490 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 03:16:05.687630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.687042 5490 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 03:16:05.687630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.687079 5490 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 03:16:05.785140 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.785107 5490 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 03:16:05.790724 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.790703 5490 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:16:05.790816 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.790739 5490 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:16:05.790816 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.790751 5490 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:16:05.790816 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.790795 5490 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:05.808032 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.807967 5490 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:05.808133 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.808039 5490 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:05.868872 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.868806 5490 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal"] Oct 31 03:16:05.869479 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.869423 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal" event={"ID":"fc792e750c2606b177de944a501bf339","Type":"ContainerStarted","Data":"859d43288fd5c20f6be76c8cf240c6a2524de8216694875ea2cef8256195811c"} Oct 31 03:16:05.869593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.869484 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal" event={"ID":"fc792e750c2606b177de944a501bf339","Type":"ContainerStarted","Data":"9464aa9bf3b2aa43daa2a9323f4fe8a5b4c275c88cedd1e89755a98a3df10be7"} Oct 31 03:16:05.869593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.869506 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" event={"ID":"9dcc8bbdd4365b42723f25cd49f22a1c","Type":"ContainerStarted","Data":"6763aef3ce6951abf3c15c898b2ccade39d013de1abd8e370e797065dfb2eba3"} Oct 31 03:16:05.869593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.869526 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" event={"ID":"9dcc8bbdd4365b42723f25cd49f22a1c","Type":"ContainerDied","Data":"392fe58da98574e342eb491e1c359bbb7928bc4a5be06969278d16c476b7f327"} Oct 31 03:16:05.869593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.869535 5490 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:05.869593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.869546 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" event={"ID":"9dcc8bbdd4365b42723f25cd49f22a1c","Type":"ContainerStarted","Data":"ab39d17504c1ed393c39b9ff8d73ce1229730f78105dc51229457d973aaf7a67"} Oct 31 03:16:05.869815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.869798 5490 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:05.886221 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.886199 5490 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:16:05.886351 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:05.886245 5490 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:05.886482 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.886466 5490 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:16:05.886523 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:05.886504 5490 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:05.958291 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.958254 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dcc8bbdd4365b42723f25cd49f22a1c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal\" (UID: \"9dcc8bbdd4365b42723f25cd49f22a1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:05.958291 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.958297 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc792e750c2606b177de944a501bf339-config\") pod \"kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal\" (UID: \"fc792e750c2606b177de944a501bf339\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:05.958488 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:05.958314 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9dcc8bbdd4365b42723f25cd49f22a1c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal\" (UID: \"9dcc8bbdd4365b42723f25cd49f22a1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:06.058933 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.058845 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc792e750c2606b177de944a501bf339-config\") pod \"kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal\" (UID: \"fc792e750c2606b177de944a501bf339\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:06.058933 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.058889 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9dcc8bbdd4365b42723f25cd49f22a1c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal\" (UID: \"9dcc8bbdd4365b42723f25cd49f22a1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:06.058933 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.058910 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dcc8bbdd4365b42723f25cd49f22a1c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal\" (UID: \"9dcc8bbdd4365b42723f25cd49f22a1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:06.059125 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.058957 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dcc8bbdd4365b42723f25cd49f22a1c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal\" (UID: \"9dcc8bbdd4365b42723f25cd49f22a1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:06.059125 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.058958 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc792e750c2606b177de944a501bf339-config\") pod \"kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal\" (UID: \"fc792e750c2606b177de944a501bf339\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:06.059125 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.058967 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9dcc8bbdd4365b42723f25cd49f22a1c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal\" (UID: \"9dcc8bbdd4365b42723f25cd49f22a1c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:06.526477 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.526440 5490 apiserver.go:52] "Watching apiserver" Oct 31 03:16:06.536857 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.536827 5490 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 03:16:06.537528 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.537508 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8","openshift-network-operator/iptables-alerter-frjws","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb","openshift-insights/insights-runtime-extractor-zngp8","openshift-cluster-node-tuning-operator/tuned-4hwhg","openshift-monitoring/prometheus-operator-9cd89b74c-968x8","openshift-monitoring/thanos-querier-b949d9bf6-8b6xc","openshift-multus/multus-additional-cni-plugins-zcfc2","openshift-network-diagnostics/network-check-target-4tzpt","openshift-monitoring/node-exporter-mtd46","openshift-multus/network-metrics-daemon-lkp2n","openshift-monitoring/prometheus-k8s-0","openshift-ovn-kubernetes/ovnkube-node-x6rwl","kube-system/global-pull-secret-syncer-gkf5m","openshift-ingress-canary/ingress-canary-vh2gt","kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal","openshift-dns/node-resolver-z74z2","kube-system/konnectivity-agent-5lds2","openshift-multus/multus-h9wtm","openshift-dns/dns-default-vcg5q","openshift-image-registry/node-ca-g2mdn"] Oct 31 03:16:06.548284 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.548256 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 03:16:06.548284 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.548272 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 03:16:06.548520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.548255 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 03:16:06.548520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.548356 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 03:16:06.548520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.548474 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 03:16:06.548520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.548474 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 03:16:06.548782 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.548766 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 03:16:06.548833 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.548768 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 03:16:06.548865 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.548834 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 03:16:06.548918 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.548904 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 03:16:06.549298 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.549281 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:16:06.549456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.549442 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 03:16:06.558357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.558335 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 03:16:06.558482 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.558386 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:16:06.558859 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.558842 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 03:16:06.559122 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.559100 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 03:16:06.559122 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.559104 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 03:16:06.559277 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.559151 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 03:16:06.559277 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.559205 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 03:16:06.559277 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.559206 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 03:16:06.559277 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.559224 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 03:16:06.559277 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.559263 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 03:16:06.559503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.559413 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 03:16:06.559503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.559428 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 03:16:06.559503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.559488 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 03:16:06.559772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.559756 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 03:16:06.559886 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.559874 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 03:16:06.559961 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.559945 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 03:16:06.560015 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.559968 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 03:16:06.560015 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.559978 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 03:16:06.560015 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.559988 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 03:16:06.560455 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.560439 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 03:16:06.560550 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.560533 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 03:16:06.560597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.560490 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 03:16:06.560597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.560525 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 03:16:06.560597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.560490 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 03:16:06.561541 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.561524 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 03:16:06.561626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.561542 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 03:16:06.561626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.561553 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7a31b67-a1b9-4b6a-a112-69cd5c833019-system-cni-dir\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:16:06.561626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.561582 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16b948f8-e3da-4285-8342-27cbd2dbd141-hosts-file\") pod \"node-resolver-z74z2\" (UID: \"16b948f8-e3da-4285-8342-27cbd2dbd141\") " pod="openshift-dns/node-resolver-z74z2" Oct 31 03:16:06.561810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.561634 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-node-log\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.561810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.561651 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-etc-selinux\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:16:06.561810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.561716 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8b2g\" (UniqueName: \"kubernetes.io/projected/571b679b-fad2-423d-96e5-8d5550bb536d-kube-api-access-r8b2g\") pod \"iptables-alerter-frjws\" (UID: \"571b679b-fad2-423d-96e5-8d5550bb536d\") " pod="openshift-network-operator/iptables-alerter-frjws" Oct 31 03:16:06.561810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.561733 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-sysctl-d\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.561810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.561756 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-sysctl-conf\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.561810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.561779 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-run\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.562150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.561815 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-multus-cni-dir\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.562150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.561858 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.562150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.561898 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 03:16:06.562150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.561904 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z82fk\" (UniqueName: \"kubernetes.io/projected/55f1a18a-59ef-4146-a0e8-3fc632ab20f5-kube-api-access-z82fk\") pod \"node-ca-g2mdn\" (UID: \"55f1a18a-59ef-4146-a0e8-3fc632ab20f5\") " pod="openshift-image-registry/node-ca-g2mdn" Oct 31 03:16:06.562150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.561959 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k8hk8\"" Oct 31 03:16:06.562150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.561954 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7a31b67-a1b9-4b6a-a112-69cd5c833019-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:16:06.562150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.561991 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 03:16:06.562150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562022 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/904f42d7-69e3-462f-9055-b6b840800f13-multus-daemon-config\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.562508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562173 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqks5\" (UniqueName: \"kubernetes.io/projected/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-kube-api-access-pqks5\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.562508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562209 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7a31b67-a1b9-4b6a-a112-69cd5c833019-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:16:06.562508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562218 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/904f42d7-69e3-462f-9055-b6b840800f13-multus-daemon-config\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.562508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562213 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f247bb3-6322-43c8-bfad-f27a41e26500-agent-certs\") pod \"konnectivity-agent-5lds2\" (UID: \"8f247bb3-6322-43c8-bfad-f27a41e26500\") " pod="kube-system/konnectivity-agent-5lds2" Oct 31 03:16:06.562508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562299 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 03:16:06.562508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562298 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7a31b67-a1b9-4b6a-a112-69cd5c833019-cnibin\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:16:06.562508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562303 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 03:16:06.562508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562305 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 03:16:06.562508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562342 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-ovnkube-config\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.562508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562360 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-sys-fs\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:16:06.562508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562377 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwzkk\" (UniqueName: \"kubernetes.io/projected/16b948f8-e3da-4285-8342-27cbd2dbd141-kube-api-access-pwzkk\") pod \"node-resolver-z74z2\" (UID: \"16b948f8-e3da-4285-8342-27cbd2dbd141\") " pod="openshift-dns/node-resolver-z74z2" Oct 31 03:16:06.562508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562418 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-run-netns\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.562508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562447 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-run-multus-certs\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.562508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562478 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hnbq\" (UniqueName: \"kubernetes.io/projected/904f42d7-69e3-462f-9055-b6b840800f13-kube-api-access-7hnbq\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.563081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562528 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 03:16:06.563081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562537 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-socket-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:16:06.563081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562535 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-ovnkube-config\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.563081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562576 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-systemd\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.563081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562600 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-hostroot\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.563081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562638 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-systemd-units\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.563081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562668 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-run-ovn\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.563081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562702 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 03:16:06.563081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562708 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7a31b67-a1b9-4b6a-a112-69cd5c833019-cni-binary-copy\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:16:06.563081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562733 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/571b679b-fad2-423d-96e5-8d5550bb536d-host-slash\") pod \"iptables-alerter-frjws\" (UID: \"571b679b-fad2-423d-96e5-8d5550bb536d\") " pod="openshift-network-operator/iptables-alerter-frjws" Oct 31 03:16:06.563081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562791 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-lib-modules\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.563081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562870 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-run-k8s-cni-cncf-io\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.563081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562902 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7a31b67-a1b9-4b6a-a112-69cd5c833019-cni-binary-copy\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:16:06.563081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.562916 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-env-overrides\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.563081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563023 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-env-overrides\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.563081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563057 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/571b679b-fad2-423d-96e5-8d5550bb536d-iptables-alerter-script\") pod \"iptables-alerter-frjws\" (UID: \"571b679b-fad2-423d-96e5-8d5550bb536d\") " pod="openshift-network-operator/iptables-alerter-frjws" Oct 31 03:16:06.563081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563090 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-cnibin\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.563852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563118 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-slash\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.563852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563138 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-modprobe-d\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.563852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563167 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-system-cni-dir\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.563852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563188 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-run-netns\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.563852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563224 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-var-lib-kubelet\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.563852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563251 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 03:16:06.563852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563254 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 03:16:06.563852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563275 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-etc-kubernetes\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.563852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563295 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 03:16:06.563852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563307 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 03:16:06.563852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563314 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 31 03:16:06.563852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563308 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-cni-bin\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.563852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563255 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/571b679b-fad2-423d-96e5-8d5550bb536d-iptables-alerter-script\") pod \"iptables-alerter-frjws\" (UID: \"571b679b-fad2-423d-96e5-8d5550bb536d\") " pod="openshift-network-operator/iptables-alerter-frjws" Oct 31 03:16:06.563852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563326 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-nwvk2\"" Oct 31 03:16:06.563852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563350 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 03:16:06.563852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563354 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 03:16:06.563852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563381 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-registration-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:16:06.563852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563427 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/904f42d7-69e3-462f-9055-b6b840800f13-cni-binary-copy\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.564616 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.564155 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-log-socket\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.564616 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.563446 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 03:16:06.564616 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.564377 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 03:16:06.564616 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.564413 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gmjq8\"" Oct 31 03:16:06.564616 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.564451 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 03:16:06.564616 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.564558 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/904f42d7-69e3-462f-9055-b6b840800f13-cni-binary-copy\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.564616 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.564608 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/55f1a18a-59ef-4146-a0e8-3fc632ab20f5-serviceca\") pod \"node-ca-g2mdn\" (UID: \"55f1a18a-59ef-4146-a0e8-3fc632ab20f5\") " pod="openshift-image-registry/node-ca-g2mdn" Oct 31 03:16:06.564941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.564653 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rtvr4\"" Oct 31 03:16:06.564941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.564663 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qsw4\" (UniqueName: \"kubernetes.io/projected/d7a31b67-a1b9-4b6a-a112-69cd5c833019-kube-api-access-2qsw4\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:16:06.564941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.564718 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-sysconfig\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.564941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.564750 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-sys\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.564941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.564884 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 03:16:06.564941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.564899 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r6lm2\"" Oct 31 03:16:06.564941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.564930 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fkjgm\"" Oct 31 03:16:06.565225 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.565117 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dk5h8\"" Oct 31 03:16:06.565271 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.565231 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-var-lib-kubelet\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.565350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.565332 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nklpg\"" Oct 31 03:16:06.565400 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.565354 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 03:16:06.565478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.565410 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 03:16:06.565478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.565423 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7a31b67-a1b9-4b6a-a112-69cd5c833019-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:16:06.565478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.565465 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7a31b67-a1b9-4b6a-a112-69cd5c833019-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:16:06.565610 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.565498 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-var-lib-cni-multus\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.565610 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.565412 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zf8rk\"" Oct 31 03:16:06.565610 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.565531 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c47g\" (UniqueName: \"kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g\") pod \"network-check-target-4tzpt\" (UID: \"d92bdf4f-f9e4-424c-9c87-ec015574ff1a\") " pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:16:06.565610 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.565564 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-run-openvswitch\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.565823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.565630 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a303dr7af9aru\"" Oct 31 03:16:06.565823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.565739 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 03:16:06.565823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.565805 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qg7r7\"" Oct 31 03:16:06.565951 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.565237 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 03:16:06.565951 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.565864 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-j2pvp\"" Oct 31 03:16:06.566170 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.566140 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rfxnp\"" Oct 31 03:16:06.566246 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.566196 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 03:16:06.566352 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.566332 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7a31b67-a1b9-4b6a-a112-69cd5c833019-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:16:06.566418 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.566359 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/55f1a18a-59ef-4146-a0e8-3fc632ab20f5-serviceca\") pod \"node-ca-g2mdn\" (UID: \"55f1a18a-59ef-4146-a0e8-3fc632ab20f5\") " pod="openshift-image-registry/node-ca-g2mdn" Oct 31 03:16:06.566418 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.566376 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-run-ovn-kubernetes\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.566653 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.566633 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-ovn-node-metrics-cert\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.566743 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.566695 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4ttf\" (UniqueName: \"kubernetes.io/projected/400b676a-a3a0-459d-9da6-8d30b2d3b497-kube-api-access-n4ttf\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:16:06.566796 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.566777 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55f1a18a-59ef-4146-a0e8-3fc632ab20f5-host\") pod \"node-ca-g2mdn\" (UID: \"55f1a18a-59ef-4146-a0e8-3fc632ab20f5\") " pod="openshift-image-registry/node-ca-g2mdn" Oct 31 03:16:06.566857 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.566809 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-tmp\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.566857 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.566835 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 03:16:06.566857 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.566844 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16b948f8-e3da-4285-8342-27cbd2dbd141-tmp-dir\") pod \"node-resolver-z74z2\" (UID: \"16b948f8-e3da-4285-8342-27cbd2dbd141\") " pod="openshift-dns/node-resolver-z74z2" Oct 31 03:16:06.566990 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.566882 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-cni-netd\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.567066 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567047 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16b948f8-e3da-4285-8342-27cbd2dbd141-tmp-dir\") pod \"node-resolver-z74z2\" (UID: \"16b948f8-e3da-4285-8342-27cbd2dbd141\") " pod="openshift-dns/node-resolver-z74z2" Oct 31 03:16:06.567138 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567098 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-tmp\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.567187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567173 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-ovnkube-script-lib\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.567242 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567210 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-tuned\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.567290 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567244 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkr86\" (UniqueName: \"kubernetes.io/projected/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-kube-api-access-dkr86\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.567369 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567344 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-multus-socket-dir-parent\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.567447 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567390 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-kubelet\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.567447 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567433 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-run-systemd\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.567590 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567471 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f247bb3-6322-43c8-bfad-f27a41e26500-konnectivity-ca\") pod \"konnectivity-agent-5lds2\" (UID: \"8f247bb3-6322-43c8-bfad-f27a41e26500\") " pod="kube-system/konnectivity-agent-5lds2" Oct 31 03:16:06.567590 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567511 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-var-lib-cni-bin\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.567590 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567545 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-etc-openvswitch\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.567590 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567355 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-tuned\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.567866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567605 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-ovnkube-script-lib\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.567866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567666 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:16:06.567866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567700 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 03:16:06.567866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567716 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-device-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:16:06.567866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567751 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-kubernetes\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.567866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567782 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-host\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.567866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567843 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-multus-conf-dir\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.568353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567888 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-var-lib-openvswitch\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.568353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.567947 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f247bb3-6322-43c8-bfad-f27a41e26500-konnectivity-ca\") pod \"konnectivity-agent-5lds2\" (UID: \"8f247bb3-6322-43c8-bfad-f27a41e26500\") " pod="kube-system/konnectivity-agent-5lds2" Oct 31 03:16:06.568353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.568192 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-ovn-node-metrics-cert\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.568353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.568303 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 03:16:06.568353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.568340 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 03:16:06.568613 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.568359 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2mhh0emo4u5nu\"" Oct 31 03:16:06.568613 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.568404 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-p6hhn\"" Oct 31 03:16:06.568613 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.568421 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-htgcs\"" Oct 31 03:16:06.568613 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.568486 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fn5cc\"" Oct 31 03:16:06.568613 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.568543 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dct2q\"" Oct 31 03:16:06.568848 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.568622 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-b55zh\"" Oct 31 03:16:06.568848 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.568631 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5s7vx\"" Oct 31 03:16:06.568848 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.568695 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p2pq6\"" Oct 31 03:16:06.568848 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.568732 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 31 03:16:06.568848 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.568798 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d5pxc\"" Oct 31 03:16:06.568848 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.568796 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 03:16:06.570942 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.570925 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 03:16:06.571322 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.571306 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 03:16:06.571739 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.571724 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqks5\" (UniqueName: \"kubernetes.io/projected/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-kube-api-access-pqks5\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.571778 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.571742 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwzkk\" (UniqueName: \"kubernetes.io/projected/16b948f8-e3da-4285-8342-27cbd2dbd141-kube-api-access-pwzkk\") pod \"node-resolver-z74z2\" (UID: \"16b948f8-e3da-4285-8342-27cbd2dbd141\") " pod="openshift-dns/node-resolver-z74z2" Oct 31 03:16:06.572316 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.572299 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z82fk\" (UniqueName: \"kubernetes.io/projected/55f1a18a-59ef-4146-a0e8-3fc632ab20f5-kube-api-access-z82fk\") pod \"node-ca-g2mdn\" (UID: \"55f1a18a-59ef-4146-a0e8-3fc632ab20f5\") " pod="openshift-image-registry/node-ca-g2mdn" Oct 31 03:16:06.572382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.572338 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hnbq\" (UniqueName: \"kubernetes.io/projected/904f42d7-69e3-462f-9055-b6b840800f13-kube-api-access-7hnbq\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.572517 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.572500 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f247bb3-6322-43c8-bfad-f27a41e26500-agent-certs\") pod \"konnectivity-agent-5lds2\" (UID: \"8f247bb3-6322-43c8-bfad-f27a41e26500\") " pod="kube-system/konnectivity-agent-5lds2" Oct 31 03:16:06.572747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.572732 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8b2g\" (UniqueName: \"kubernetes.io/projected/571b679b-fad2-423d-96e5-8d5550bb536d-kube-api-access-r8b2g\") pod \"iptables-alerter-frjws\" (UID: \"571b679b-fad2-423d-96e5-8d5550bb536d\") " pod="openshift-network-operator/iptables-alerter-frjws" Oct 31 03:16:06.573920 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.573904 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qsw4\" (UniqueName: \"kubernetes.io/projected/d7a31b67-a1b9-4b6a-a112-69cd5c833019-kube-api-access-2qsw4\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:16:06.577635 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.577618 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkr86\" (UniqueName: \"kubernetes.io/projected/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-kube-api-access-dkr86\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.578297 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.578275 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4ttf\" (UniqueName: \"kubernetes.io/projected/400b676a-a3a0-459d-9da6-8d30b2d3b497-kube-api-access-n4ttf\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:16:06.578565 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.578551 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c47g\" (UniqueName: \"kubernetes.io/projected/d92bdf4f-f9e4-424c-9c87-ec015574ff1a-kube-api-access-7c47g\") pod \"network-check-target-4tzpt\" (UID: \"d92bdf4f-f9e4-424c-9c87-ec015574ff1a\") " pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:16:06.626102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.626073 5490 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:06.626257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.626181 5490 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:06.638133 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.638104 5490 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:16:06.638275 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:06.638219 5490 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:06.638597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.638578 5490 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 03:16:06.638654 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:06.638627 5490 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal" Oct 31 03:16:06.643959 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.643926 5490 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 03:16:06.668532 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.668505 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7a31b67-a1b9-4b6a-a112-69cd5c833019-system-cni-dir\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:16:06.668694 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.668538 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-node-log\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.668694 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.668562 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-etc-selinux\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:16:06.668694 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.668609 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7a31b67-a1b9-4b6a-a112-69cd5c833019-system-cni-dir\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:16:06.668694 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.668625 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-etc-selinux\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:16:06.668694 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.668611 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-node-log\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.668694 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.668668 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/575aa82a-6d21-46c1-80ff-7f756f00c108-sys\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:06.669029 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.668741 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:06.669029 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.668767 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-multus-cni-dir\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.669029 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.668792 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w596\" (UniqueName: \"kubernetes.io/projected/08da34c7-c69a-4eb9-848a-399cab14840e-kube-api-access-2w596\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:16:06.669029 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.668812 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:16:06.669029 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.668827 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/062e4d3d-4460-4412-93d3-22681a9a74e6-config-volume\") pod \"dns-default-vcg5q\" (UID: \"062e4d3d-4460-4412-93d3-22681a9a74e6\") " pod="openshift-dns/dns-default-vcg5q" Oct 31 03:16:06.669029 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.668845 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.669029 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.668878 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-multus-cni-dir\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.669029 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.668894 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-web-config\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.669029 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.668951 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-socket-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:16:06.669029 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.668983 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-wtmp\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:06.669029 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669008 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-config-out\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.669029 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669032 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-hostroot\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669035 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669058 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/571b679b-fad2-423d-96e5-8d5550bb536d-host-slash\") pod \"iptables-alerter-frjws\" (UID: \"571b679b-fad2-423d-96e5-8d5550bb536d\") " pod="openshift-network-operator/iptables-alerter-frjws" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669028 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669110 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-hostroot\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669134 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-lib-modules\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669151 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-config-out\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669153 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50818995-4b65-457e-9b55-250b548e80d3-cert\") pod \"ingress-canary-vh2gt\" (UID: \"50818995-4b65-457e-9b55-250b548e80d3\") " pod="openshift-ingress-canary/ingress-canary-vh2gt" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669200 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-run-k8s-cni-cncf-io\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669205 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-socket-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669213 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-web-config\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669228 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-tls\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669246 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-run-k8s-cni-cncf-io\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669284 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/571b679b-fad2-423d-96e5-8d5550bb536d-host-slash\") pod \"iptables-alerter-frjws\" (UID: \"571b679b-fad2-423d-96e5-8d5550bb536d\") " pod="openshift-network-operator/iptables-alerter-frjws" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669279 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-lib-modules\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669321 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669385 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669431 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50818995-4b65-457e-9b55-250b548e80d3-cert\") pod \"ingress-canary-vh2gt\" (UID: \"50818995-4b65-457e-9b55-250b548e80d3\") " pod="openshift-ingress-canary/ingress-canary-vh2gt" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669448 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-tls\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669452 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/062e4d3d-4460-4412-93d3-22681a9a74e6-config-volume\") pod \"dns-default-vcg5q\" (UID: \"062e4d3d-4460-4412-93d3-22681a9a74e6\") " pod="openshift-dns/dns-default-vcg5q" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669500 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs\") pod \"network-metrics-daemon-lkp2n\" (UID: \"d680db8a-15e4-4817-bff1-a420f9f54681\") " pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669510 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.669538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669539 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-slash\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669642 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-slash\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669646 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-modprobe-d\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669711 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08da34c7-c69a-4eb9-848a-399cab14840e-data-volume\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669756 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669772 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d680db8a-15e4-4817-bff1-a420f9f54681-metrics-certs\") pod \"network-metrics-daemon-lkp2n\" (UID: \"d680db8a-15e4-4817-bff1-a420f9f54681\") " pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669782 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-system-cni-dir\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669805 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-modprobe-d\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669818 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/08da34c7-c69a-4eb9-848a-399cab14840e-data-volume\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669814 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-run-netns\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669856 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-system-cni-dir\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669888 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-var-lib-kubelet\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669910 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-run-netns\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669916 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-var-lib-kubelet\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669949 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bkq\" (UniqueName: \"kubernetes.io/projected/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-kube-api-access-q8bkq\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669966 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-log-socket\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.669983 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-sys\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670014 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-var-lib-kubelet\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670034 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-log-socket\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670065 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-var-lib-kubelet\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670069 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-sys\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670085 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbnp\" (UniqueName: \"kubernetes.io/projected/062e4d3d-4460-4412-93d3-22681a9a74e6-kube-api-access-kgbnp\") pod \"dns-default-vcg5q\" (UID: \"062e4d3d-4460-4412-93d3-22681a9a74e6\") " pod="openshift-dns/dns-default-vcg5q" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670102 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq8hk\" (UniqueName: \"kubernetes.io/projected/8bae2714-9c1b-44db-a6ef-b0b008cf1fa2-kube-api-access-jq8hk\") pod \"monitoring-plugin-77f4f76594-5kbd8\" (UID: \"8bae2714-9c1b-44db-a6ef-b0b008cf1fa2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670125 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062e4d3d-4460-4412-93d3-22681a9a74e6-tmp-dir\") pod \"dns-default-vcg5q\" (UID: \"062e4d3d-4460-4412-93d3-22681a9a74e6\") " pod="openshift-dns/dns-default-vcg5q" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670148 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670173 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670198 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/062e4d3d-4460-4412-93d3-22681a9a74e6-tmp-dir\") pod \"dns-default-vcg5q\" (UID: \"062e4d3d-4460-4412-93d3-22681a9a74e6\") " pod="openshift-dns/dns-default-vcg5q" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670203 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670240 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55f1a18a-59ef-4146-a0e8-3fc632ab20f5-host\") pod \"node-ca-g2mdn\" (UID: \"55f1a18a-59ef-4146-a0e8-3fc632ab20f5\") " pod="openshift-image-registry/node-ca-g2mdn" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670266 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-tls\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670291 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670302 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670317 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-cni-netd\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670346 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670349 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55f1a18a-59ef-4146-a0e8-3fc632ab20f5-host\") pod \"node-ca-g2mdn\" (UID: \"55f1a18a-59ef-4146-a0e8-3fc632ab20f5\") " pod="openshift-image-registry/node-ca-g2mdn" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670384 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-cni-netd\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670390 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:16:06.670429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670416 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-multus-socket-dir-parent\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670456 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-kubelet\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670496 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08da34c7-c69a-4eb9-848a-399cab14840e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670503 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670515 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-multus-socket-dir-parent\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670521 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg9j9\" (UniqueName: \"kubernetes.io/projected/575aa82a-6d21-46c1-80ff-7f756f00c108-kube-api-access-pg9j9\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670536 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-kubelet\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670536 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-tls\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670558 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-grpc-tls\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670517 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670623 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-var-lib-cni-bin\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670650 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-etc-openvswitch\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670698 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/08da34c7-c69a-4eb9-848a-399cab14840e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670666 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-device-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670736 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-kubernetes\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670746 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-var-lib-cni-bin\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670750 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-etc-openvswitch\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670765 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08da34c7-c69a-4eb9-848a-399cab14840e-crio-socket\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670794 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-kubernetes\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670795 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-device-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670804 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-grpc-tls\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670807 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d877fe5-7b94-4645-bff5-6df0796bb637-dbus\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670839 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w65q\" (UniqueName: \"kubernetes.io/projected/d680db8a-15e4-4817-bff1-a420f9f54681-kube-api-access-6w65q\") pod \"network-metrics-daemon-lkp2n\" (UID: \"d680db8a-15e4-4817-bff1-a420f9f54681\") " pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670855 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-multus-conf-dir\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670870 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670890 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-multus-conf-dir\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670912 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670975 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16b948f8-e3da-4285-8342-27cbd2dbd141-hosts-file\") pod \"node-resolver-z74z2\" (UID: \"16b948f8-e3da-4285-8342-27cbd2dbd141\") " pod="openshift-dns/node-resolver-z74z2" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.670993 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-sysctl-d\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671033 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-sysctl-conf\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671065 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-run\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671092 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-sysctl-d\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671092 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/062e4d3d-4460-4412-93d3-22681a9a74e6-metrics-tls\") pod \"dns-default-vcg5q\" (UID: \"062e4d3d-4460-4412-93d3-22681a9a74e6\") " pod="openshift-dns/dns-default-vcg5q" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671112 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16b948f8-e3da-4285-8342-27cbd2dbd141-hosts-file\") pod \"node-resolver-z74z2\" (UID: \"16b948f8-e3da-4285-8342-27cbd2dbd141\") " pod="openshift-dns/node-resolver-z74z2" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671128 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-run\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671140 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-tmp\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671153 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-sysctl-conf\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671166 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k6ph\" (UniqueName: \"kubernetes.io/projected/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-kube-api-access-6k6ph\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671193 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-accelerators-collector-config\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671206 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-tmp\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671222 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671254 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkbnh\" (UniqueName: \"kubernetes.io/projected/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-kube-api-access-vkbnh\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671266 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671279 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671307 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7a31b67-a1b9-4b6a-a112-69cd5c833019-cnibin\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671353 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/062e4d3d-4460-4412-93d3-22681a9a74e6-metrics-tls\") pod \"dns-default-vcg5q\" (UID: \"062e4d3d-4460-4412-93d3-22681a9a74e6\") " pod="openshift-dns/dns-default-vcg5q" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671361 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7a31b67-a1b9-4b6a-a112-69cd5c833019-cnibin\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671394 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-accelerators-collector-config\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671404 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-textfile\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671446 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-metrics-client-ca\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671473 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-config\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671495 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-textfile\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671498 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671528 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8bae2714-9c1b-44db-a6ef-b0b008cf1fa2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5kbd8\" (UID: \"8bae2714-9c1b-44db-a6ef-b0b008cf1fa2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671540 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671552 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671582 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671610 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-sys-fs\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671628 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-metrics-client-ca\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671634 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-run-netns\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671669 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-run-netns\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671700 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-run-multus-certs\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671706 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-config\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671727 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-systemd\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671750 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671755 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671801 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-sys-fs\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671806 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8bae2714-9c1b-44db-a6ef-b0b008cf1fa2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5kbd8\" (UID: \"8bae2714-9c1b-44db-a6ef-b0b008cf1fa2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671809 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671757 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-run-multus-certs\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671824 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-systemd\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671837 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-systemd-units\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671861 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-run-ovn\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671888 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d877fe5-7b94-4645-bff5-6df0796bb637-kubelet-config\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671898 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-systemd-units\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671897 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671916 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/575aa82a-6d21-46c1-80ff-7f756f00c108-metrics-client-ca\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671921 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7d877fe5-7b94-4645-bff5-6df0796bb637-original-pull-secret\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671921 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-run-ovn\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671942 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671974 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.671988 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672002 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-cnibin\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672030 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672058 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-etc-kubernetes\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672082 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-cni-bin\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672087 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-cnibin\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672101 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/575aa82a-6d21-46c1-80ff-7f756f00c108-metrics-client-ca\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:06.672057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672110 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-registration-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672140 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672144 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672159 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-etc-kubernetes\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672150 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672181 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-sysconfig\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672183 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672210 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-cni-bin\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672210 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08da34c7-c69a-4eb9-848a-399cab14840e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672256 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-registration-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672259 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-etc-sysconfig\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672299 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672312 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8ncx\" (UniqueName: \"kubernetes.io/projected/50818995-4b65-457e-9b55-250b548e80d3-kube-api-access-b8ncx\") pod \"ingress-canary-vh2gt\" (UID: \"50818995-4b65-457e-9b55-250b548e80d3\") " pod="openshift-ingress-canary/ingress-canary-vh2gt" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672344 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7a31b67-a1b9-4b6a-a112-69cd5c833019-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672372 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672377 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672413 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-var-lib-cni-multus\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672461 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7a31b67-a1b9-4b6a-a112-69cd5c833019-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zcfc2\" (UID: \"d7a31b67-a1b9-4b6a-a112-69cd5c833019\") " pod="openshift-multus/multus-additional-cni-plugins-zcfc2" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672478 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/08da34c7-c69a-4eb9-848a-399cab14840e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672495 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-run-openvswitch\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672518 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672526 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-run-ovn-kubernetes\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672530 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-run-openvswitch\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672481 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/904f42d7-69e3-462f-9055-b6b840800f13-host-var-lib-cni-multus\") pod \"multus-h9wtm\" (UID: \"904f42d7-69e3-462f-9055-b6b840800f13\") " pod="openshift-multus/multus-h9wtm" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672559 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672565 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-host-run-ovn-kubernetes\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672579 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672601 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-run-systemd\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672620 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/575aa82a-6d21-46c1-80ff-7f756f00c108-root\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672635 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672651 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672666 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-host\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672724 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ad1f3a7-eea4-4104-b3ff-e860da35bf9c-host\") pod \"tuned-4hwhg\" (UID: \"7ad1f3a7-eea4-4104-b3ff-e860da35bf9c\") " pod="openshift-cluster-node-tuning-operator/tuned-4hwhg" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672738 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672747 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-var-lib-openvswitch\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672756 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672766 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672779 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-run-systemd\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672808 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/92cf7508-2fc5-49a5-9e71-5d9c73bf200e-var-lib-openvswitch\") pod \"ovnkube-node-x6rwl\" (UID: \"92cf7508-2fc5-49a5-9e71-5d9c73bf200e\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672842 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/400b676a-a3a0-459d-9da6-8d30b2d3b497-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ttmkb\" (UID: \"400b676a-a3a0-459d-9da6-8d30b2d3b497\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672925 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:16:06.674652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.672971 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.676249 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.676232 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w596\" (UniqueName: \"kubernetes.io/projected/08da34c7-c69a-4eb9-848a-399cab14840e-kube-api-access-2w596\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:16:06.678579 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.678562 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg9j9\" (UniqueName: \"kubernetes.io/projected/575aa82a-6d21-46c1-80ff-7f756f00c108-kube-api-access-pg9j9\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:06.678761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.678746 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgbnp\" (UniqueName: \"kubernetes.io/projected/062e4d3d-4460-4412-93d3-22681a9a74e6-kube-api-access-kgbnp\") pod \"dns-default-vcg5q\" (UID: \"062e4d3d-4460-4412-93d3-22681a9a74e6\") " pod="openshift-dns/dns-default-vcg5q" Oct 31 03:16:06.680287 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.680268 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8ncx\" (UniqueName: \"kubernetes.io/projected/50818995-4b65-457e-9b55-250b548e80d3-kube-api-access-b8ncx\") pod \"ingress-canary-vh2gt\" (UID: \"50818995-4b65-457e-9b55-250b548e80d3\") " pod="openshift-ingress-canary/ingress-canary-vh2gt" Oct 31 03:16:06.680721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.680700 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq8hk\" (UniqueName: \"kubernetes.io/projected/8bae2714-9c1b-44db-a6ef-b0b008cf1fa2-kube-api-access-jq8hk\") pod \"monitoring-plugin-77f4f76594-5kbd8\" (UID: \"8bae2714-9c1b-44db-a6ef-b0b008cf1fa2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" Oct 31 03:16:06.680765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.680731 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkbnh\" (UniqueName: \"kubernetes.io/projected/d395aca5-b8ff-4da1-ad31-dd26e2427c2e-kube-api-access-vkbnh\") pod \"thanos-querier-b949d9bf6-8b6xc\" (UID: \"d395aca5-b8ff-4da1-ad31-dd26e2427c2e\") " pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:16:06.680806 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.680786 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k6ph\" (UniqueName: \"kubernetes.io/projected/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-kube-api-access-6k6ph\") pod \"prometheus-k8s-0\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:06.680846 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.680832 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w65q\" (UniqueName: \"kubernetes.io/projected/d680db8a-15e4-4817-bff1-a420f9f54681-kube-api-access-6w65q\") pod \"network-metrics-daemon-lkp2n\" (UID: \"d680db8a-15e4-4817-bff1-a420f9f54681\") " pod="openshift-multus/network-metrics-daemon-lkp2n" Oct 31 03:16:06.680884 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.680868 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8bkq\" (UniqueName: \"kubernetes.io/projected/cf66bb31-fee6-49fd-aaea-c8e8d8361c77-kube-api-access-q8bkq\") pod \"prometheus-operator-9cd89b74c-968x8\" (UID: \"cf66bb31-fee6-49fd-aaea-c8e8d8361c77\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-968x8" Oct 31 03:16:06.773205 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.773167 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08da34c7-c69a-4eb9-848a-399cab14840e-crio-socket\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:16:06.773205 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.773207 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d877fe5-7b94-4645-bff5-6df0796bb637-dbus\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:16:06.773401 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.773237 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d877fe5-7b94-4645-bff5-6df0796bb637-kubelet-config\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:16:06.773401 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.773272 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/575aa82a-6d21-46c1-80ff-7f756f00c108-root\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:06.773401 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.773292 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/575aa82a-6d21-46c1-80ff-7f756f00c108-sys\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:06.773401 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.773319 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-wtmp\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:06.773401 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.773316 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/08da34c7-c69a-4eb9-848a-399cab14840e-crio-socket\") pod \"insights-runtime-extractor-zngp8\" (UID: \"08da34c7-c69a-4eb9-848a-399cab14840e\") " pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:16:06.773401 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.773316 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7d877fe5-7b94-4645-bff5-6df0796bb637-dbus\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:16:06.773401 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.773356 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/575aa82a-6d21-46c1-80ff-7f756f00c108-sys\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:06.773401 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.773382 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7d877fe5-7b94-4645-bff5-6df0796bb637-kubelet-config\") pod \"global-pull-secret-syncer-gkf5m\" (UID: \"7d877fe5-7b94-4645-bff5-6df0796bb637\") " pod="kube-system/global-pull-secret-syncer-gkf5m" Oct 31 03:16:06.773401 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.773391 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/575aa82a-6d21-46c1-80ff-7f756f00c108-root\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:06.773694 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:06.773423 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/575aa82a-6d21-46c1-80ff-7f756f00c108-node-exporter-wtmp\") pod \"node-exporter-mtd46\" (UID: \"575aa82a-6d21-46c1-80ff-7f756f00c108\") " pod="openshift-monitoring/node-exporter-mtd46" Oct 31 03:16:07.228142 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:07.228107 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" Oct 31 03:16:07.930204 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:07.930173 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:08.238269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:08.238172 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:08.257853 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:08.257827 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:08.404359 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:08.404325 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:08.449971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:08.449937 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:16:08.621805 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:08.621719 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vcg5q" Oct 31 03:16:08.636709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:08.635995 5490 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 03:16:10.362404 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:10.362364 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:10.439154 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:10.439117 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:16:10.515109 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:10.515072 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:10.583027 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:10.582999 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6rwl" Oct 31 03:16:12.555944 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:12.555898 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:12.712587 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:12.712552 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:13.996966 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:13.996934 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5lds2" Oct 31 03:16:14.928387 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:14.928351 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:15.096507 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:15.096463 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:17.026987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:17.026900 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:17.126949 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:17.126911 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:16:18.833285 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:18.833243 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:20.008806 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:20.008769 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:20.194791 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:20.194747 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:26.583037 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:26.583000 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:26.716965 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:26.716924 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:31.917503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:31.917465 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 03:16:31.918106 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:31.918038 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="prometheus" containerID="cri-o://f09ca5e1f3c887b5375b981e2890e138df11d33248e20b2b13bf9d75c52e5166" gracePeriod=600 Oct 31 03:16:31.918347 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:31.918078 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="kube-rbac-proxy" containerID="cri-o://1ef4fa90eee8f33863f63f8d99682f28211953f3423beb11b161cee799829e17" gracePeriod=600 Oct 31 03:16:31.918476 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:31.918109 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="kube-rbac-proxy-web" containerID="cri-o://70dc7ed0f447a3f58f9f85ab5b6592af61e08fe846c746d1ed4e6c152d509099" gracePeriod=600 Oct 31 03:16:31.918476 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:31.918089 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="kube-rbac-proxy-thanos" containerID="cri-o://60b9ceab13d0559e2a1a78023b130560057961a4e0e000c4387f52fc81acff90" gracePeriod=600 Oct 31 03:16:31.918587 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:31.918114 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="thanos-sidecar" containerID="cri-o://e9b8514fdc995e8fcd0a477419c170cb1908c1e61f48e3f434e11052201af26a" gracePeriod=600 Oct 31 03:16:31.918587 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:31.918119 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="config-reloader" containerID="cri-o://9af6916b470d852a952ad50fd11465a94577599a53e39f1372d4623509a94a9b" gracePeriod=600 Oct 31 03:16:32.731362 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:32.731330 5490 generic.go:356] "Generic (PLEG): container finished" podID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerID="60b9ceab13d0559e2a1a78023b130560057961a4e0e000c4387f52fc81acff90" exitCode=0 Oct 31 03:16:32.731362 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:32.731353 5490 generic.go:356] "Generic (PLEG): container finished" podID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerID="1ef4fa90eee8f33863f63f8d99682f28211953f3423beb11b161cee799829e17" exitCode=0 Oct 31 03:16:32.731362 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:32.731359 5490 generic.go:356] "Generic (PLEG): container finished" podID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerID="70dc7ed0f447a3f58f9f85ab5b6592af61e08fe846c746d1ed4e6c152d509099" exitCode=0 Oct 31 03:16:32.731362 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:32.731364 5490 generic.go:356] "Generic (PLEG): container finished" podID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerID="e9b8514fdc995e8fcd0a477419c170cb1908c1e61f48e3f434e11052201af26a" exitCode=0 Oct 31 03:16:32.731362 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:32.731369 5490 generic.go:356] "Generic (PLEG): container finished" podID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerID="9af6916b470d852a952ad50fd11465a94577599a53e39f1372d4623509a94a9b" exitCode=0 Oct 31 03:16:32.731777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:32.731404 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74b3277a-2966-4f41-99cf-1a1de0ba9d0a","Type":"ContainerDied","Data":"60b9ceab13d0559e2a1a78023b130560057961a4e0e000c4387f52fc81acff90"} Oct 31 03:16:32.731777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:32.731489 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74b3277a-2966-4f41-99cf-1a1de0ba9d0a","Type":"ContainerDied","Data":"1ef4fa90eee8f33863f63f8d99682f28211953f3423beb11b161cee799829e17"} Oct 31 03:16:32.731777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:32.731527 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74b3277a-2966-4f41-99cf-1a1de0ba9d0a","Type":"ContainerDied","Data":"70dc7ed0f447a3f58f9f85ab5b6592af61e08fe846c746d1ed4e6c152d509099"} Oct 31 03:16:32.731777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:32.731561 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74b3277a-2966-4f41-99cf-1a1de0ba9d0a","Type":"ContainerDied","Data":"e9b8514fdc995e8fcd0a477419c170cb1908c1e61f48e3f434e11052201af26a"} Oct 31 03:16:32.731777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:32.731599 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74b3277a-2966-4f41-99cf-1a1de0ba9d0a","Type":"ContainerDied","Data":"9af6916b470d852a952ad50fd11465a94577599a53e39f1372d4623509a94a9b"} Oct 31 03:16:33.258019 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:33.257968 5490 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="f09ca5e1f3c887b5375b981e2890e138df11d33248e20b2b13bf9d75c52e5166" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 31 03:16:33.258491 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:33.258037 5490 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="f09ca5e1f3c887b5375b981e2890e138df11d33248e20b2b13bf9d75c52e5166" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 31 03:16:33.258491 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:33.258060 5490 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="f09ca5e1f3c887b5375b981e2890e138df11d33248e20b2b13bf9d75c52e5166" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 31 03:16:33.258491 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:33.258080 5490 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled" Oct 31 03:16:35.287019 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:35.286962 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:35.408814 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:35.408778 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:35.628818 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:35.627497 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:35.830729 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:35.830671 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:36.722610 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:36.722573 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:38.258658 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:38.258605 5490 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="f09ca5e1f3c887b5375b981e2890e138df11d33248e20b2b13bf9d75c52e5166" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 31 03:16:38.259069 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:38.258667 5490 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="f09ca5e1f3c887b5375b981e2890e138df11d33248e20b2b13bf9d75c52e5166" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 31 03:16:38.259069 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:38.258710 5490 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="f09ca5e1f3c887b5375b981e2890e138df11d33248e20b2b13bf9d75c52e5166" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 31 03:16:38.259069 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:38.258727 5490 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled" Oct 31 03:16:38.452538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:38.452501 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4tzpt" Oct 31 03:16:39.460435 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.460410 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.635316 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.635231 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-tls-assets\" (UniqueName: \"kubernetes.io/projected/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-tls-assets\") pod \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " Oct 31 03:16:39.635316 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.635278 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-thanos-prometheus-http-client-file\") pod \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " Oct 31 03:16:39.635316 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.635307 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-trusted-ca-bundle\") pod \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " Oct 31 03:16:39.635664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.635337 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-0\") pod \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " Oct 31 03:16:39.635664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.635386 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-kube-rbac-proxy\") pod \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " Oct 31 03:16:39.635664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.635411 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-grpc-tls\") pod \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " Oct 31 03:16:39.635664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.635445 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " Oct 31 03:16:39.635664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.635563 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-tls\") pod \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " Oct 31 03:16:39.635664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.635611 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-metrics-client-ca\") pod \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " Oct 31 03:16:39.635664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.635643 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-web-config\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-web-config\") pod \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " Oct 31 03:16:39.636043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.635711 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-metrics-client-certs\") pod \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " Oct 31 03:16:39.636043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.635747 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " Oct 31 03:16:39.636043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.635810 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-1\") pod \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " Oct 31 03:16:39.636043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.635840 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-kubelet-serving-ca-bundle\") pod \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " Oct 31 03:16:39.636043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.635859 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-trusted-ca-bundle" pod "74b3277a-2966-4f41-99cf-1a1de0ba9d0a" (UID: "74b3277a-2966-4f41-99cf-1a1de0ba9d0a"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:16:39.636043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.635873 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-db\") pod \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " Oct 31 03:16:39.636043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.635906 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-serving-certs-ca-bundle\") pod \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " Oct 31 03:16:39.636043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.635942 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-config\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-config\") pod \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " Oct 31 03:16:39.636043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.635981 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-kube-api-access-6k6ph\" (UniqueName: \"kubernetes.io/projected/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-kube-api-access-6k6ph\") pod \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " Oct 31 03:16:39.636043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.636036 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-2\") pod \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " Oct 31 03:16:39.636502 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.636075 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-config-out\" (UniqueName: \"kubernetes.io/empty-dir/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-config-out\") pod \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\" (UID: \"74b3277a-2966-4f41-99cf-1a1de0ba9d0a\") " Oct 31 03:16:39.636502 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.636293 5490 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.636735 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.636702 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-metrics-client-ca" pod "74b3277a-2966-4f41-99cf-1a1de0ba9d0a" (UID: "74b3277a-2966-4f41-99cf-1a1de0ba9d0a"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:16:39.637300 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.637017 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-0" pod "74b3277a-2966-4f41-99cf-1a1de0ba9d0a" (UID: "74b3277a-2966-4f41-99cf-1a1de0ba9d0a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:16:39.638327 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.638231 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-db" pod "74b3277a-2966-4f41-99cf-1a1de0ba9d0a" (UID: "74b3277a-2966-4f41-99cf-1a1de0ba9d0a"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:16:39.638732 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.638479 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-1" pod "74b3277a-2966-4f41-99cf-1a1de0ba9d0a" (UID: "74b3277a-2966-4f41-99cf-1a1de0ba9d0a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:16:39.639006 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.638969 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-kubelet-serving-ca-bundle" pod "74b3277a-2966-4f41-99cf-1a1de0ba9d0a" (UID: "74b3277a-2966-4f41-99cf-1a1de0ba9d0a"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:16:39.639085 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.638990 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-2" pod "74b3277a-2966-4f41-99cf-1a1de0ba9d0a" (UID: "74b3277a-2966-4f41-99cf-1a1de0ba9d0a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:16:39.639143 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.639119 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-config-out" pod "74b3277a-2966-4f41-99cf-1a1de0ba9d0a" (UID: "74b3277a-2966-4f41-99cf-1a1de0ba9d0a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:16:39.639268 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.639233 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-serving-certs-ca-bundle" pod "74b3277a-2966-4f41-99cf-1a1de0ba9d0a" (UID: "74b3277a-2966-4f41-99cf-1a1de0ba9d0a"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:16:39.639384 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.639337 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-thanos-prometheus-http-client-file" pod "74b3277a-2966-4f41-99cf-1a1de0ba9d0a" (UID: "74b3277a-2966-4f41-99cf-1a1de0ba9d0a"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:16:39.639564 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.639535 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-grpc-tls" pod "74b3277a-2966-4f41-99cf-1a1de0ba9d0a" (UID: "74b3277a-2966-4f41-99cf-1a1de0ba9d0a"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:16:39.639889 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.639767 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-metrics-client-certs" pod "74b3277a-2966-4f41-99cf-1a1de0ba9d0a" (UID: "74b3277a-2966-4f41-99cf-1a1de0ba9d0a"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:16:39.639889 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.639841 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-tls" pod "74b3277a-2966-4f41-99cf-1a1de0ba9d0a" (UID: "74b3277a-2966-4f41-99cf-1a1de0ba9d0a"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:16:39.640065 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.639986 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-tls-assets" pod "74b3277a-2966-4f41-99cf-1a1de0ba9d0a" (UID: "74b3277a-2966-4f41-99cf-1a1de0ba9d0a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:16:39.640122 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.640073 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-thanos-sidecar-tls" pod "74b3277a-2966-4f41-99cf-1a1de0ba9d0a" (UID: "74b3277a-2966-4f41-99cf-1a1de0ba9d0a"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:16:39.640588 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.640561 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-config" pod "74b3277a-2966-4f41-99cf-1a1de0ba9d0a" (UID: "74b3277a-2966-4f41-99cf-1a1de0ba9d0a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:16:39.640869 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.640840 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-kube-rbac-proxy-web" pod "74b3277a-2966-4f41-99cf-1a1de0ba9d0a" (UID: "74b3277a-2966-4f41-99cf-1a1de0ba9d0a"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:16:39.640972 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.640882 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-kube-rbac-proxy" pod "74b3277a-2966-4f41-99cf-1a1de0ba9d0a" (UID: "74b3277a-2966-4f41-99cf-1a1de0ba9d0a"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:16:39.641382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.641353 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-kube-api-access-6k6ph" pod "74b3277a-2966-4f41-99cf-1a1de0ba9d0a" (UID: "74b3277a-2966-4f41-99cf-1a1de0ba9d0a"). InnerVolumeSpecName "kube-api-access-6k6ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:16:39.651348 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.651321 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-web-config" pod "74b3277a-2966-4f41-99cf-1a1de0ba9d0a" (UID: "74b3277a-2966-4f41-99cf-1a1de0ba9d0a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:16:39.736902 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.736866 5490 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-kube-rbac-proxy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.736902 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.736898 5490 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-grpc-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.737097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.736911 5490 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.737097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.736923 5490 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.737097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.736933 5490 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-metrics-client-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.737097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.736942 5490 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-web-config\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.737097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.736951 5490 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-metrics-client-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.737097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.736960 5490 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.737097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.736969 5490 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.737097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.736979 5490 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.737097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.736987 5490 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-db\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.737097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.736996 5490 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.737097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.737005 5490 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-config\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.737097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.737014 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k6ph\" (UniqueName: \"kubernetes.io/projected/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-kube-api-access-6k6ph\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.737097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.737022 5490 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.737097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.737030 5490 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-config-out\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.737097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.737038 5490 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-tls-assets\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.737097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.737047 5490 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.737097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.737055 5490 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/74b3277a-2966-4f41-99cf-1a1de0ba9d0a-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:16:39.756533 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.756501 5490 generic.go:356] "Generic (PLEG): container finished" podID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerID="f09ca5e1f3c887b5375b981e2890e138df11d33248e20b2b13bf9d75c52e5166" exitCode=0 Oct 31 03:16:39.756692 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.756549 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74b3277a-2966-4f41-99cf-1a1de0ba9d0a","Type":"ContainerDied","Data":"f09ca5e1f3c887b5375b981e2890e138df11d33248e20b2b13bf9d75c52e5166"} Oct 31 03:16:39.756692 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.756581 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.756692 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.756607 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"74b3277a-2966-4f41-99cf-1a1de0ba9d0a","Type":"ContainerDied","Data":"eee132e0769005a7f31e3d3b4c1f929f8d96f2bc565566ef9b26c3b9bfaf298a"} Oct 31 03:16:39.756692 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.756641 5490 scope.go:122] "RemoveContainer" containerID="60b9ceab13d0559e2a1a78023b130560057961a4e0e000c4387f52fc81acff90" Oct 31 03:16:39.764663 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.764643 5490 scope.go:122] "RemoveContainer" containerID="1ef4fa90eee8f33863f63f8d99682f28211953f3423beb11b161cee799829e17" Oct 31 03:16:39.771328 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.771310 5490 scope.go:122] "RemoveContainer" containerID="70dc7ed0f447a3f58f9f85ab5b6592af61e08fe846c746d1ed4e6c152d509099" Oct 31 03:16:39.777560 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.777543 5490 scope.go:122] "RemoveContainer" containerID="e9b8514fdc995e8fcd0a477419c170cb1908c1e61f48e3f434e11052201af26a" Oct 31 03:16:39.778708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.778690 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 03:16:39.784496 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.784475 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 03:16:39.784971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.784958 5490 scope.go:122] "RemoveContainer" containerID="9af6916b470d852a952ad50fd11465a94577599a53e39f1372d4623509a94a9b" Oct 31 03:16:39.791403 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.791388 5490 scope.go:122] "RemoveContainer" containerID="f09ca5e1f3c887b5375b981e2890e138df11d33248e20b2b13bf9d75c52e5166" Oct 31 03:16:39.798258 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.798241 5490 scope.go:122] "RemoveContainer" containerID="945c8971b228a65db2916c2b8a4ab87372c601d5d632272b26611e95c031c386" Oct 31 03:16:39.804589 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.804575 5490 scope.go:122] "RemoveContainer" containerID="60b9ceab13d0559e2a1a78023b130560057961a4e0e000c4387f52fc81acff90" Oct 31 03:16:39.807123 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:39.807101 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60b9ceab13d0559e2a1a78023b130560057961a4e0e000c4387f52fc81acff90\": container with ID starting with 60b9ceab13d0559e2a1a78023b130560057961a4e0e000c4387f52fc81acff90 not found: ID does not exist" containerID="60b9ceab13d0559e2a1a78023b130560057961a4e0e000c4387f52fc81acff90" Oct 31 03:16:39.807210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.807133 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b9ceab13d0559e2a1a78023b130560057961a4e0e000c4387f52fc81acff90"} err="failed to get container status \"60b9ceab13d0559e2a1a78023b130560057961a4e0e000c4387f52fc81acff90\": rpc error: code = NotFound desc = could not find container \"60b9ceab13d0559e2a1a78023b130560057961a4e0e000c4387f52fc81acff90\": container with ID starting with 60b9ceab13d0559e2a1a78023b130560057961a4e0e000c4387f52fc81acff90 not found: ID does not exist" Oct 31 03:16:39.807210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.807172 5490 scope.go:122] "RemoveContainer" containerID="1ef4fa90eee8f33863f63f8d99682f28211953f3423beb11b161cee799829e17" Oct 31 03:16:39.807558 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:39.807543 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef4fa90eee8f33863f63f8d99682f28211953f3423beb11b161cee799829e17\": container with ID starting with 1ef4fa90eee8f33863f63f8d99682f28211953f3423beb11b161cee799829e17 not found: ID does not exist" containerID="1ef4fa90eee8f33863f63f8d99682f28211953f3423beb11b161cee799829e17" Oct 31 03:16:39.807620 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.807571 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef4fa90eee8f33863f63f8d99682f28211953f3423beb11b161cee799829e17"} err="failed to get container status \"1ef4fa90eee8f33863f63f8d99682f28211953f3423beb11b161cee799829e17\": rpc error: code = NotFound desc = could not find container \"1ef4fa90eee8f33863f63f8d99682f28211953f3423beb11b161cee799829e17\": container with ID starting with 1ef4fa90eee8f33863f63f8d99682f28211953f3423beb11b161cee799829e17 not found: ID does not exist" Oct 31 03:16:39.807620 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.807585 5490 scope.go:122] "RemoveContainer" containerID="70dc7ed0f447a3f58f9f85ab5b6592af61e08fe846c746d1ed4e6c152d509099" Oct 31 03:16:39.807874 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:39.807852 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70dc7ed0f447a3f58f9f85ab5b6592af61e08fe846c746d1ed4e6c152d509099\": container with ID starting with 70dc7ed0f447a3f58f9f85ab5b6592af61e08fe846c746d1ed4e6c152d509099 not found: ID does not exist" containerID="70dc7ed0f447a3f58f9f85ab5b6592af61e08fe846c746d1ed4e6c152d509099" Oct 31 03:16:39.807932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.807880 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70dc7ed0f447a3f58f9f85ab5b6592af61e08fe846c746d1ed4e6c152d509099"} err="failed to get container status \"70dc7ed0f447a3f58f9f85ab5b6592af61e08fe846c746d1ed4e6c152d509099\": rpc error: code = NotFound desc = could not find container \"70dc7ed0f447a3f58f9f85ab5b6592af61e08fe846c746d1ed4e6c152d509099\": container with ID starting with 70dc7ed0f447a3f58f9f85ab5b6592af61e08fe846c746d1ed4e6c152d509099 not found: ID does not exist" Oct 31 03:16:39.807932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.807896 5490 scope.go:122] "RemoveContainer" containerID="e9b8514fdc995e8fcd0a477419c170cb1908c1e61f48e3f434e11052201af26a" Oct 31 03:16:39.808156 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:39.808138 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b8514fdc995e8fcd0a477419c170cb1908c1e61f48e3f434e11052201af26a\": container with ID starting with e9b8514fdc995e8fcd0a477419c170cb1908c1e61f48e3f434e11052201af26a not found: ID does not exist" containerID="e9b8514fdc995e8fcd0a477419c170cb1908c1e61f48e3f434e11052201af26a" Oct 31 03:16:39.808211 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.808163 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b8514fdc995e8fcd0a477419c170cb1908c1e61f48e3f434e11052201af26a"} err="failed to get container status \"e9b8514fdc995e8fcd0a477419c170cb1908c1e61f48e3f434e11052201af26a\": rpc error: code = NotFound desc = could not find container \"e9b8514fdc995e8fcd0a477419c170cb1908c1e61f48e3f434e11052201af26a\": container with ID starting with e9b8514fdc995e8fcd0a477419c170cb1908c1e61f48e3f434e11052201af26a not found: ID does not exist" Oct 31 03:16:39.808211 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.808177 5490 scope.go:122] "RemoveContainer" containerID="9af6916b470d852a952ad50fd11465a94577599a53e39f1372d4623509a94a9b" Oct 31 03:16:39.808406 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:39.808390 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af6916b470d852a952ad50fd11465a94577599a53e39f1372d4623509a94a9b\": container with ID starting with 9af6916b470d852a952ad50fd11465a94577599a53e39f1372d4623509a94a9b not found: ID does not exist" containerID="9af6916b470d852a952ad50fd11465a94577599a53e39f1372d4623509a94a9b" Oct 31 03:16:39.808455 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.808418 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af6916b470d852a952ad50fd11465a94577599a53e39f1372d4623509a94a9b"} err="failed to get container status \"9af6916b470d852a952ad50fd11465a94577599a53e39f1372d4623509a94a9b\": rpc error: code = NotFound desc = could not find container \"9af6916b470d852a952ad50fd11465a94577599a53e39f1372d4623509a94a9b\": container with ID starting with 9af6916b470d852a952ad50fd11465a94577599a53e39f1372d4623509a94a9b not found: ID does not exist" Oct 31 03:16:39.808455 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.808430 5490 scope.go:122] "RemoveContainer" containerID="f09ca5e1f3c887b5375b981e2890e138df11d33248e20b2b13bf9d75c52e5166" Oct 31 03:16:39.808669 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:39.808655 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f09ca5e1f3c887b5375b981e2890e138df11d33248e20b2b13bf9d75c52e5166\": container with ID starting with f09ca5e1f3c887b5375b981e2890e138df11d33248e20b2b13bf9d75c52e5166 not found: ID does not exist" containerID="f09ca5e1f3c887b5375b981e2890e138df11d33248e20b2b13bf9d75c52e5166" Oct 31 03:16:39.808733 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.808696 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f09ca5e1f3c887b5375b981e2890e138df11d33248e20b2b13bf9d75c52e5166"} err="failed to get container status \"f09ca5e1f3c887b5375b981e2890e138df11d33248e20b2b13bf9d75c52e5166\": rpc error: code = NotFound desc = could not find container \"f09ca5e1f3c887b5375b981e2890e138df11d33248e20b2b13bf9d75c52e5166\": container with ID starting with f09ca5e1f3c887b5375b981e2890e138df11d33248e20b2b13bf9d75c52e5166 not found: ID does not exist" Oct 31 03:16:39.808733 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.808710 5490 scope.go:122] "RemoveContainer" containerID="945c8971b228a65db2916c2b8a4ab87372c601d5d632272b26611e95c031c386" Oct 31 03:16:39.808937 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:39.808921 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"945c8971b228a65db2916c2b8a4ab87372c601d5d632272b26611e95c031c386\": container with ID starting with 945c8971b228a65db2916c2b8a4ab87372c601d5d632272b26611e95c031c386 not found: ID does not exist" containerID="945c8971b228a65db2916c2b8a4ab87372c601d5d632272b26611e95c031c386" Oct 31 03:16:39.808975 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.808942 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945c8971b228a65db2916c2b8a4ab87372c601d5d632272b26611e95c031c386"} err="failed to get container status \"945c8971b228a65db2916c2b8a4ab87372c601d5d632272b26611e95c031c386\": rpc error: code = NotFound desc = could not find container \"945c8971b228a65db2916c2b8a4ab87372c601d5d632272b26611e95c031c386\": container with ID starting with 945c8971b228a65db2916c2b8a4ab87372c601d5d632272b26611e95c031c386 not found: ID does not exist" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810361 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810645 5490 cpu_manager.go:414] "containerMap: removing container" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="kube-rbac-proxy" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810658 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="kube-rbac-proxy" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810666 5490 cpu_manager.go:414] "containerMap: removing container" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="kube-rbac-proxy-web" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810671 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="kube-rbac-proxy-web" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810698 5490 cpu_manager.go:414] "containerMap: removing container" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="config-reloader" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810704 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="config-reloader" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810712 5490 cpu_manager.go:414] "containerMap: removing container" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="prometheus" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810717 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="prometheus" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810730 5490 cpu_manager.go:414] "containerMap: removing container" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="kube-rbac-proxy-thanos" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810735 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="kube-rbac-proxy-thanos" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810742 5490 cpu_manager.go:414] "containerMap: removing container" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="thanos-sidecar" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810746 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="thanos-sidecar" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810751 5490 cpu_manager.go:414] "containerMap: removing container" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="init-config-reloader" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810756 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="init-config-reloader" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810803 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="config-reloader" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810810 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="prometheus" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810817 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="kube-rbac-proxy-thanos" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810822 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="thanos-sidecar" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810828 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="init-config-reloader" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810843 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="kube-rbac-proxy" Oct 31 03:16:39.810868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.810848 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" containerName="kube-rbac-proxy-web" Oct 31 03:16:39.816871 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.816851 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.821033 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.821011 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 03:16:39.821033 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.821025 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 03:16:39.821215 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.821198 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 03:16:39.821264 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.821205 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 03:16:39.821264 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.821253 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 03:16:39.822078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.822001 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 03:16:39.822078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.822037 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 03:16:39.822078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.822050 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 03:16:39.822078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.822061 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 03:16:39.822078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.822077 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-p6hhn\"" Oct 31 03:16:39.822353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.822177 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 03:16:39.822353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.822239 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 03:16:39.822353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.822305 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 03:16:39.822353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.822340 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a303dr7af9aru\"" Oct 31 03:16:39.822522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.822369 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 03:16:39.826016 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.825994 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 03:16:39.826121 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.826106 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 03:16:39.845053 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.845027 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 03:16:39.938557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.938521 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-config\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.938557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.938561 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.938848 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.938577 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-web-config\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.938848 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.938714 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.938848 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.938756 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c835066f-39c8-42b4-9f99-1055495bcc69-config-out\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.938848 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.938774 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c835066f-39c8-42b4-9f99-1055495bcc69-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.939051 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.938838 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.939051 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.938918 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.939051 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.938981 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c835066f-39c8-42b4-9f99-1055495bcc69-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.939051 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.939008 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.939187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.939062 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.939187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.939088 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.939187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.939114 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.939187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.939138 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.939187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.939159 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.939353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.939211 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.939353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.939249 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.939353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.939289 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.939353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.939306 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrfl8\" (UniqueName: \"kubernetes.io/projected/c835066f-39c8-42b4-9f99-1055495bcc69-kube-api-access-qrfl8\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:39.939353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:39.939341 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.040570 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.040527 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.040570 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.040573 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.040860 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.040595 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.040860 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.040626 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.040860 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.040647 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrfl8\" (UniqueName: \"kubernetes.io/projected/c835066f-39c8-42b4-9f99-1055495bcc69-kube-api-access-qrfl8\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.040860 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.040671 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.040860 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.040721 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-config\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.040860 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.040798 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.040860 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.040839 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-web-config\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.041187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.040872 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.041187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.040911 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c835066f-39c8-42b4-9f99-1055495bcc69-config-out\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.041187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.040938 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c835066f-39c8-42b4-9f99-1055495bcc69-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.041187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.040961 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.041187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.040993 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.041187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.041018 5490 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 03:16:40.041187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.041046 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c835066f-39c8-42b4-9f99-1055495bcc69-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.041187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.041080 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.041187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.041107 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.041187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.041135 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.041187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.041178 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.041720 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.041216 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.041720 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.041429 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.042004 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.041956 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.043885 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.043581 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c835066f-39c8-42b4-9f99-1055495bcc69-config-out\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.043885 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.043653 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.044057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.043959 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c835066f-39c8-42b4-9f99-1055495bcc69-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.044057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.044038 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-web-config\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.044057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.044046 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-config\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.044926 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.044270 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.044926 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.044344 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c835066f-39c8-42b4-9f99-1055495bcc69-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.044926 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.044649 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.044926 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.044843 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.044926 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.044908 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.045244 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.045120 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.045358 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.045307 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c835066f-39c8-42b4-9f99-1055495bcc69-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.045646 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.045620 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.046379 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.046360 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.046445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.046360 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.047070 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.047047 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.047151 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.047081 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c835066f-39c8-42b4-9f99-1055495bcc69-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.048792 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.048776 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrfl8\" (UniqueName: \"kubernetes.io/projected/c835066f-39c8-42b4-9f99-1055495bcc69-kube-api-access-qrfl8\") pod \"prometheus-k8s-0\" (UID: \"c835066f-39c8-42b4-9f99-1055495bcc69\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.139895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.139852 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:40.424059 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.424024 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 03:16:40.428013 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:16:40.427988 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc835066f_39c8_42b4_9f99_1055495bcc69.slice/crio-69cc097615d5853032c9bbde6db6e6a396c2b9a7e4acb80b56d5d1aa80edf643 WatchSource:0}: Error finding container 69cc097615d5853032c9bbde6db6e6a396c2b9a7e4acb80b56d5d1aa80edf643: Status 404 returned error can't find the container with id 69cc097615d5853032c9bbde6db6e6a396c2b9a7e4acb80b56d5d1aa80edf643 Oct 31 03:16:40.762670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.762582 5490 generic.go:356] "Generic (PLEG): container finished" podID="c835066f-39c8-42b4-9f99-1055495bcc69" containerID="c61581d6eae523d3700596aadfd2e483297a9aaf1d1cf041fc68a423986162a2" exitCode=0 Oct 31 03:16:40.762670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.762639 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c835066f-39c8-42b4-9f99-1055495bcc69","Type":"ContainerDied","Data":"c61581d6eae523d3700596aadfd2e483297a9aaf1d1cf041fc68a423986162a2"} Oct 31 03:16:40.763100 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:40.762728 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c835066f-39c8-42b4-9f99-1055495bcc69","Type":"ContainerStarted","Data":"69cc097615d5853032c9bbde6db6e6a396c2b9a7e4acb80b56d5d1aa80edf643"} Oct 31 03:16:41.572785 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:41.572709 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74b3277a-2966-4f41-99cf-1a1de0ba9d0a" path="/var/lib/kubelet/pods/74b3277a-2966-4f41-99cf-1a1de0ba9d0a/volumes" Oct 31 03:16:41.771624 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:41.771586 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c835066f-39c8-42b4-9f99-1055495bcc69","Type":"ContainerStarted","Data":"b4531ef04699d930aad1ff3289a3cca18477fa56653a3bb82359eae92bfe842b"} Oct 31 03:16:41.772042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:41.771656 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c835066f-39c8-42b4-9f99-1055495bcc69","Type":"ContainerStarted","Data":"717b6a122b02f1da905a8a32730f2cd99397aaefbd62d5bb638b67cce9fd892f"} Oct 31 03:16:41.772042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:41.771711 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c835066f-39c8-42b4-9f99-1055495bcc69","Type":"ContainerStarted","Data":"84bdcab2d2e17eb9129effb97acd7ba9f0d2c21eb52e01db475ffbcafae3d082"} Oct 31 03:16:41.772042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:41.771742 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c835066f-39c8-42b4-9f99-1055495bcc69","Type":"ContainerStarted","Data":"83fb0f712513ddb0f6a9bbceb8defd0387a9c3df7a40c5566cda52c1f32d58ce"} Oct 31 03:16:41.772042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:41.771766 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c835066f-39c8-42b4-9f99-1055495bcc69","Type":"ContainerStarted","Data":"e45b39b28d48508bc0cbd5182aa719005bcd234f120cb445000444910d7d77ec"} Oct 31 03:16:41.772042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:41.771797 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c835066f-39c8-42b4-9f99-1055495bcc69","Type":"ContainerStarted","Data":"03b7939b3b56af0e43222d53088648cd2930043f2c665662d97be3e259276221"} Oct 31 03:16:41.808022 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:41.807946 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.791500982 podStartE2EDuration="2.807928871s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.427889ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:41.80673276 +0000 UTC m=+36.341427714" watchObservedRunningTime="2026-10-31 03:16:41.807928871 +0000 UTC m=+36.342623828" Oct 31 03:16:42.649210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:42.649178 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vh2gt_50818995-4b65-457e-9b55-250b548e80d3/serve-healthcheck-canary/0.log" Oct 31 03:16:43.049177 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:43.049141 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:43.179534 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:43.179501 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:16:45.140496 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:16:45.140457 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:16:47.125804 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:16:47.125767 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:17:11.009484 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:17:11.009446 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:17:11.025547 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:17:11.025518 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:17:11.896724 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:17:11.896694 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:17:14.977093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:17:14.977055 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:17.126263 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:17:17.126226 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:17:21.799078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:17:21.799043 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:21.917453 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:17:21.917410 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:34.542095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:17:34.542057 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:34.671457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:17:34.671423 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:38.393139 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:17:38.393104 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:38.516895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:17:38.516862 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:40.059124 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:17:40.059091 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:40.182317 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:17:40.182281 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:47.125383 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:17:47.125299 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:17:50.020341 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:17:50.020307 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:17:50.147389 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:17:50.147353 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:09.887442 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:18:09.887408 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:10.011610 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:18:10.011567 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:15.019864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:18:15.019820 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:17.125287 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:18:17.125249 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:18:38.824737 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:18:38.824699 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:41.786084 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:18:41.786045 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:41.911494 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:18:41.911465 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:18:47.125647 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:18:47.125609 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:19:15.085877 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:19:15.085843 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:17.126134 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:19:17.126051 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:19:47.127849 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:19:47.127801 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:19:54.801415 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:19:54.801382 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:19:56.069409 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:19:56.069374 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:03.571578 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:03.571395 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:03.719958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:03.719919 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:03.892525 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:03.892436 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:05.221928 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:05.221893 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:05.357203 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:05.357174 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:05.960327 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:05.960293 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p"] Oct 31 03:20:05.963606 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:05.963589 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p" Oct 31 03:20:05.966882 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:05.966858 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-kjqfs\"" Oct 31 03:20:05.967886 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:05.967866 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 03:20:05.967886 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:05.967879 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 03:20:05.971821 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:05.971803 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p"] Oct 31 03:20:06.099462 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.099427 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18ef0547-2e2f-44ef-87dd-e307a41d6f32-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p\" (UID: \"18ef0547-2e2f-44ef-87dd-e307a41d6f32\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p" Oct 31 03:20:06.099716 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.099566 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18ef0547-2e2f-44ef-87dd-e307a41d6f32-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p\" (UID: \"18ef0547-2e2f-44ef-87dd-e307a41d6f32\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p" Oct 31 03:20:06.099716 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.099606 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvwbt\" (UniqueName: \"kubernetes.io/projected/18ef0547-2e2f-44ef-87dd-e307a41d6f32-kube-api-access-rvwbt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p\" (UID: \"18ef0547-2e2f-44ef-87dd-e307a41d6f32\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p" Oct 31 03:20:06.100235 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.100204 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6"] Oct 31 03:20:06.103611 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.103596 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9"] Oct 31 03:20:06.103771 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.103756 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6" Oct 31 03:20:06.106833 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.106813 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9" Oct 31 03:20:06.111762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.111737 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9"] Oct 31 03:20:06.121043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.121020 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-5q2bp\"" Oct 31 03:20:06.121154 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.121101 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-c7mnl\"" Oct 31 03:20:06.122743 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.117071 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6"] Oct 31 03:20:06.161290 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.161259 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn"] Oct 31 03:20:06.164487 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.164471 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn" Oct 31 03:20:06.173641 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.173620 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-ssqh5\"" Oct 31 03:20:06.174530 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.174510 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn"] Oct 31 03:20:06.200765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.200730 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4lt9\" (UniqueName: \"kubernetes.io/projected/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-kube-api-access-j4lt9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6\" (UID: \"7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6" Oct 31 03:20:06.200765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.200768 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18ef0547-2e2f-44ef-87dd-e307a41d6f32-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p\" (UID: \"18ef0547-2e2f-44ef-87dd-e307a41d6f32\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p" Oct 31 03:20:06.200983 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.200791 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6\" (UID: \"7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6" Oct 31 03:20:06.200983 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.200906 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6\" (UID: \"7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6" Oct 31 03:20:06.201059 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.201006 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18ef0547-2e2f-44ef-87dd-e307a41d6f32-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p\" (UID: \"18ef0547-2e2f-44ef-87dd-e307a41d6f32\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p" Oct 31 03:20:06.201059 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.201025 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvwbt\" (UniqueName: \"kubernetes.io/projected/18ef0547-2e2f-44ef-87dd-e307a41d6f32-kube-api-access-rvwbt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p\" (UID: \"18ef0547-2e2f-44ef-87dd-e307a41d6f32\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p" Oct 31 03:20:06.201147 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.201130 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18ef0547-2e2f-44ef-87dd-e307a41d6f32-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p\" (UID: \"18ef0547-2e2f-44ef-87dd-e307a41d6f32\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p" Oct 31 03:20:06.201344 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.201327 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18ef0547-2e2f-44ef-87dd-e307a41d6f32-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p\" (UID: \"18ef0547-2e2f-44ef-87dd-e307a41d6f32\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p" Oct 31 03:20:06.209324 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.209290 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvwbt\" (UniqueName: \"kubernetes.io/projected/18ef0547-2e2f-44ef-87dd-e307a41d6f32-kube-api-access-rvwbt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p\" (UID: \"18ef0547-2e2f-44ef-87dd-e307a41d6f32\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p" Oct 31 03:20:06.257611 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.257529 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm"] Oct 31 03:20:06.261093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.261072 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm" Oct 31 03:20:06.264141 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.264120 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-vgth7\"" Oct 31 03:20:06.269210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.269189 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm"] Oct 31 03:20:06.279211 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.279186 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p" Oct 31 03:20:06.301370 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.301343 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msjr6\" (UniqueName: \"kubernetes.io/projected/8e5abc82-cd8d-4810-911a-fbe67b573695-kube-api-access-msjr6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9\" (UID: \"8e5abc82-cd8d-4810-911a-fbe67b573695\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9" Oct 31 03:20:06.301483 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.301384 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e5abc82-cd8d-4810-911a-fbe67b573695-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9\" (UID: \"8e5abc82-cd8d-4810-911a-fbe67b573695\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9" Oct 31 03:20:06.301483 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.301406 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a67f04ce-194b-4404-9180-8076e089c10c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn\" (UID: \"a67f04ce-194b-4404-9180-8076e089c10c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn" Oct 31 03:20:06.301572 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.301551 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4lt9\" (UniqueName: \"kubernetes.io/projected/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-kube-api-access-j4lt9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6\" (UID: \"7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6" Oct 31 03:20:06.301663 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.301593 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e5abc82-cd8d-4810-911a-fbe67b573695-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9\" (UID: \"8e5abc82-cd8d-4810-911a-fbe67b573695\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9" Oct 31 03:20:06.301663 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.301640 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6\" (UID: \"7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6" Oct 31 03:20:06.301893 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.301669 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a67f04ce-194b-4404-9180-8076e089c10c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn\" (UID: \"a67f04ce-194b-4404-9180-8076e089c10c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn" Oct 31 03:20:06.301893 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.301723 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6\" (UID: \"7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6" Oct 31 03:20:06.301893 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.301757 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88ggk\" (UniqueName: \"kubernetes.io/projected/a67f04ce-194b-4404-9180-8076e089c10c-kube-api-access-88ggk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn\" (UID: \"a67f04ce-194b-4404-9180-8076e089c10c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn" Oct 31 03:20:06.302208 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.302190 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6\" (UID: \"7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6" Oct 31 03:20:06.302312 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.302258 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6\" (UID: \"7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6" Oct 31 03:20:06.309793 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.309767 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4lt9\" (UniqueName: \"kubernetes.io/projected/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-kube-api-access-j4lt9\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6\" (UID: \"7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6" Oct 31 03:20:06.402910 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.402871 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e5abc82-cd8d-4810-911a-fbe67b573695-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9\" (UID: \"8e5abc82-cd8d-4810-911a-fbe67b573695\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9" Oct 31 03:20:06.403105 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.402928 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a67f04ce-194b-4404-9180-8076e089c10c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn\" (UID: \"a67f04ce-194b-4404-9180-8076e089c10c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn" Oct 31 03:20:06.403105 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.403039 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfxk9\" (UniqueName: \"kubernetes.io/projected/3b81b232-bcdc-4208-82f7-78e7fd4674d9-kube-api-access-mfxk9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm\" (UID: \"3b81b232-bcdc-4208-82f7-78e7fd4674d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm" Oct 31 03:20:06.403105 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.403082 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88ggk\" (UniqueName: \"kubernetes.io/projected/a67f04ce-194b-4404-9180-8076e089c10c-kube-api-access-88ggk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn\" (UID: \"a67f04ce-194b-4404-9180-8076e089c10c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn" Oct 31 03:20:06.403271 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.403211 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msjr6\" (UniqueName: \"kubernetes.io/projected/8e5abc82-cd8d-4810-911a-fbe67b573695-kube-api-access-msjr6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9\" (UID: \"8e5abc82-cd8d-4810-911a-fbe67b573695\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9" Oct 31 03:20:06.403271 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.403263 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b81b232-bcdc-4208-82f7-78e7fd4674d9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm\" (UID: \"3b81b232-bcdc-4208-82f7-78e7fd4674d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm" Oct 31 03:20:06.403368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.403271 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a67f04ce-194b-4404-9180-8076e089c10c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn\" (UID: \"a67f04ce-194b-4404-9180-8076e089c10c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn" Oct 31 03:20:06.403368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.403309 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e5abc82-cd8d-4810-911a-fbe67b573695-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9\" (UID: \"8e5abc82-cd8d-4810-911a-fbe67b573695\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9" Oct 31 03:20:06.403368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.403312 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e5abc82-cd8d-4810-911a-fbe67b573695-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9\" (UID: \"8e5abc82-cd8d-4810-911a-fbe67b573695\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9" Oct 31 03:20:06.403507 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.403373 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a67f04ce-194b-4404-9180-8076e089c10c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn\" (UID: \"a67f04ce-194b-4404-9180-8076e089c10c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn" Oct 31 03:20:06.403507 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.403416 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b81b232-bcdc-4208-82f7-78e7fd4674d9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm\" (UID: \"3b81b232-bcdc-4208-82f7-78e7fd4674d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm" Oct 31 03:20:06.403603 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.403536 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e5abc82-cd8d-4810-911a-fbe67b573695-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9\" (UID: \"8e5abc82-cd8d-4810-911a-fbe67b573695\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9" Oct 31 03:20:06.403705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.403660 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a67f04ce-194b-4404-9180-8076e089c10c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn\" (UID: \"a67f04ce-194b-4404-9180-8076e089c10c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn" Oct 31 03:20:06.411067 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.411043 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88ggk\" (UniqueName: \"kubernetes.io/projected/a67f04ce-194b-4404-9180-8076e089c10c-kube-api-access-88ggk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn\" (UID: \"a67f04ce-194b-4404-9180-8076e089c10c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn" Oct 31 03:20:06.411170 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.411067 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msjr6\" (UniqueName: \"kubernetes.io/projected/8e5abc82-cd8d-4810-911a-fbe67b573695-kube-api-access-msjr6\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9\" (UID: \"8e5abc82-cd8d-4810-911a-fbe67b573695\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9" Oct 31 03:20:06.422826 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.422803 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6" Oct 31 03:20:06.430130 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.430102 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9" Oct 31 03:20:06.432428 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.432404 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:06.480260 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.479604 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn" Oct 31 03:20:06.504294 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.504256 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b81b232-bcdc-4208-82f7-78e7fd4674d9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm\" (UID: \"3b81b232-bcdc-4208-82f7-78e7fd4674d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm" Oct 31 03:20:06.504415 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.504338 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfxk9\" (UniqueName: \"kubernetes.io/projected/3b81b232-bcdc-4208-82f7-78e7fd4674d9-kube-api-access-mfxk9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm\" (UID: \"3b81b232-bcdc-4208-82f7-78e7fd4674d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm" Oct 31 03:20:06.504487 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.504412 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b81b232-bcdc-4208-82f7-78e7fd4674d9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm\" (UID: \"3b81b232-bcdc-4208-82f7-78e7fd4674d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm" Oct 31 03:20:06.505078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.504857 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b81b232-bcdc-4208-82f7-78e7fd4674d9-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm\" (UID: \"3b81b232-bcdc-4208-82f7-78e7fd4674d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm" Oct 31 03:20:06.505078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.505017 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b81b232-bcdc-4208-82f7-78e7fd4674d9-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm\" (UID: \"3b81b232-bcdc-4208-82f7-78e7fd4674d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm" Oct 31 03:20:06.514055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.513998 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfxk9\" (UniqueName: \"kubernetes.io/projected/3b81b232-bcdc-4208-82f7-78e7fd4674d9-kube-api-access-mfxk9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm\" (UID: \"3b81b232-bcdc-4208-82f7-78e7fd4674d9\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm" Oct 31 03:20:06.558703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.558661 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p"] Oct 31 03:20:06.562872 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:20:06.562844 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ef0547_2e2f_44ef_87dd_e307a41d6f32.slice/crio-a536d33c6b15ba832f8b5242b69e25406b2ab44ef4db28c27f4c322b64fad17d WatchSource:0}: Error finding container a536d33c6b15ba832f8b5242b69e25406b2ab44ef4db28c27f4c322b64fad17d: Status 404 returned error can't find the container with id a536d33c6b15ba832f8b5242b69e25406b2ab44ef4db28c27f4c322b64fad17d Oct 31 03:20:06.567048 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.567028 5490 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:20:06.578514 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.578495 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm" Oct 31 03:20:06.698492 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.698458 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6"] Oct 31 03:20:06.701627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.701603 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9"] Oct 31 03:20:06.702051 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:20:06.702027 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c48eb5a_93d1_4451_b42b_f7bbca7f3e2f.slice/crio-e2b4614d35dce556327a9642c1c6d8c59244d1e2753f4f8004a15fcac9acd81d WatchSource:0}: Error finding container e2b4614d35dce556327a9642c1c6d8c59244d1e2753f4f8004a15fcac9acd81d: Status 404 returned error can't find the container with id e2b4614d35dce556327a9642c1c6d8c59244d1e2753f4f8004a15fcac9acd81d Oct 31 03:20:06.705108 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:20:06.705046 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5abc82_cd8d_4810_911a_fbe67b573695.slice/crio-f11ac91718699b6da0f4e1b76ea587b9114a5b0b89d6f4f86301546d5a794334 WatchSource:0}: Error finding container f11ac91718699b6da0f4e1b76ea587b9114a5b0b89d6f4f86301546d5a794334: Status 404 returned error can't find the container with id f11ac91718699b6da0f4e1b76ea587b9114a5b0b89d6f4f86301546d5a794334 Oct 31 03:20:06.758949 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.758914 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn"] Oct 31 03:20:06.761791 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:20:06.761764 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67f04ce_194b_4404_9180_8076e089c10c.slice/crio-5024a1b7cc35f5037c581efcd5f34902e5a88d94beedecb4e32d2b4d1095938c WatchSource:0}: Error finding container 5024a1b7cc35f5037c581efcd5f34902e5a88d94beedecb4e32d2b4d1095938c: Status 404 returned error can't find the container with id 5024a1b7cc35f5037c581efcd5f34902e5a88d94beedecb4e32d2b4d1095938c Oct 31 03:20:06.847199 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:06.847167 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm"] Oct 31 03:20:06.850572 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:20:06.850546 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b81b232_bcdc_4208_82f7_78e7fd4674d9.slice/crio-3fbb31aac9b2c6508abe921c713bfaff06ec5ece6b9bb4c83bbdf495cd80158c WatchSource:0}: Error finding container 3fbb31aac9b2c6508abe921c713bfaff06ec5ece6b9bb4c83bbdf495cd80158c: Status 404 returned error can't find the container with id 3fbb31aac9b2c6508abe921c713bfaff06ec5ece6b9bb4c83bbdf495cd80158c Oct 31 03:20:07.437031 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:07.436990 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p" event={"ID":"18ef0547-2e2f-44ef-87dd-e307a41d6f32","Type":"ContainerStarted","Data":"a536d33c6b15ba832f8b5242b69e25406b2ab44ef4db28c27f4c322b64fad17d"} Oct 31 03:20:07.438922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:07.438852 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9" event={"ID":"8e5abc82-cd8d-4810-911a-fbe67b573695","Type":"ContainerStarted","Data":"f11ac91718699b6da0f4e1b76ea587b9114a5b0b89d6f4f86301546d5a794334"} Oct 31 03:20:07.441224 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:07.441196 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6" event={"ID":"7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f","Type":"ContainerStarted","Data":"e2b4614d35dce556327a9642c1c6d8c59244d1e2753f4f8004a15fcac9acd81d"} Oct 31 03:20:07.443202 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:07.443178 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm" event={"ID":"3b81b232-bcdc-4208-82f7-78e7fd4674d9","Type":"ContainerStarted","Data":"3fbb31aac9b2c6508abe921c713bfaff06ec5ece6b9bb4c83bbdf495cd80158c"} Oct 31 03:20:07.445515 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:07.445481 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn" event={"ID":"a67f04ce-194b-4404-9180-8076e089c10c","Type":"ContainerStarted","Data":"5024a1b7cc35f5037c581efcd5f34902e5a88d94beedecb4e32d2b4d1095938c"} Oct 31 03:20:07.757206 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:07.757128 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq"] Oct 31 03:20:07.761042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:07.761016 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq" Oct 31 03:20:07.767733 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:07.767705 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-n9pmz\"" Oct 31 03:20:07.771906 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:07.771768 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq"] Oct 31 03:20:07.916908 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:07.916857 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfnff\" (UniqueName: \"kubernetes.io/projected/5b19c45f-afd6-45eb-9a42-4afff1071032-kube-api-access-pfnff\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq\" (UID: \"5b19c45f-afd6-45eb-9a42-4afff1071032\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq" Oct 31 03:20:07.917091 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:07.916922 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b19c45f-afd6-45eb-9a42-4afff1071032-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq\" (UID: \"5b19c45f-afd6-45eb-9a42-4afff1071032\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq" Oct 31 03:20:07.917091 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:07.917000 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b19c45f-afd6-45eb-9a42-4afff1071032-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq\" (UID: \"5b19c45f-afd6-45eb-9a42-4afff1071032\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq" Oct 31 03:20:08.019582 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:08.018419 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b19c45f-afd6-45eb-9a42-4afff1071032-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq\" (UID: \"5b19c45f-afd6-45eb-9a42-4afff1071032\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq" Oct 31 03:20:08.019582 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:08.018496 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfnff\" (UniqueName: \"kubernetes.io/projected/5b19c45f-afd6-45eb-9a42-4afff1071032-kube-api-access-pfnff\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq\" (UID: \"5b19c45f-afd6-45eb-9a42-4afff1071032\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq" Oct 31 03:20:08.019582 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:08.018539 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b19c45f-afd6-45eb-9a42-4afff1071032-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq\" (UID: \"5b19c45f-afd6-45eb-9a42-4afff1071032\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq" Oct 31 03:20:08.019582 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:08.018993 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b19c45f-afd6-45eb-9a42-4afff1071032-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq\" (UID: \"5b19c45f-afd6-45eb-9a42-4afff1071032\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq" Oct 31 03:20:08.019582 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:08.019235 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b19c45f-afd6-45eb-9a42-4afff1071032-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq\" (UID: \"5b19c45f-afd6-45eb-9a42-4afff1071032\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq" Oct 31 03:20:08.035735 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:08.035670 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfnff\" (UniqueName: \"kubernetes.io/projected/5b19c45f-afd6-45eb-9a42-4afff1071032-kube-api-access-pfnff\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq\" (UID: \"5b19c45f-afd6-45eb-9a42-4afff1071032\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq" Oct 31 03:20:08.085350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:08.085311 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq" Oct 31 03:20:08.375463 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:08.375383 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq"] Oct 31 03:20:08.382605 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:20:08.382574 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b19c45f_afd6_45eb_9a42_4afff1071032.slice/crio-48db54c3d3b6ba63824c8bc32e66344e81920ad3b1777bdfdbfc794331934008 WatchSource:0}: Error finding container 48db54c3d3b6ba63824c8bc32e66344e81920ad3b1777bdfdbfc794331934008: Status 404 returned error can't find the container with id 48db54c3d3b6ba63824c8bc32e66344e81920ad3b1777bdfdbfc794331934008 Oct 31 03:20:08.454021 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:08.453630 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq" event={"ID":"5b19c45f-afd6-45eb-9a42-4afff1071032","Type":"ContainerStarted","Data":"48db54c3d3b6ba63824c8bc32e66344e81920ad3b1777bdfdbfc794331934008"} Oct 31 03:20:10.152845 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.152772 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:10.301650 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.301614 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:10.523856 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.523816 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7"] Oct 31 03:20:10.528650 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.527853 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7" Oct 31 03:20:10.532564 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.532520 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-7s88r\"" Oct 31 03:20:10.535751 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.535613 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7"] Oct 31 03:20:10.642044 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.642001 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac761636-38a9-4f92-b114-1285fc6a2442-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7\" (UID: \"ac761636-38a9-4f92-b114-1285fc6a2442\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7" Oct 31 03:20:10.642448 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.642421 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmpkl\" (UniqueName: \"kubernetes.io/projected/ac761636-38a9-4f92-b114-1285fc6a2442-kube-api-access-qmpkl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7\" (UID: \"ac761636-38a9-4f92-b114-1285fc6a2442\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7" Oct 31 03:20:10.642566 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.642552 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac761636-38a9-4f92-b114-1285fc6a2442-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7\" (UID: \"ac761636-38a9-4f92-b114-1285fc6a2442\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7" Oct 31 03:20:10.743144 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.743096 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac761636-38a9-4f92-b114-1285fc6a2442-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7\" (UID: \"ac761636-38a9-4f92-b114-1285fc6a2442\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7" Oct 31 03:20:10.743316 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.743181 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac761636-38a9-4f92-b114-1285fc6a2442-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7\" (UID: \"ac761636-38a9-4f92-b114-1285fc6a2442\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7" Oct 31 03:20:10.743316 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.743233 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmpkl\" (UniqueName: \"kubernetes.io/projected/ac761636-38a9-4f92-b114-1285fc6a2442-kube-api-access-qmpkl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7\" (UID: \"ac761636-38a9-4f92-b114-1285fc6a2442\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7" Oct 31 03:20:10.744050 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.744021 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac761636-38a9-4f92-b114-1285fc6a2442-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7\" (UID: \"ac761636-38a9-4f92-b114-1285fc6a2442\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7" Oct 31 03:20:10.744298 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.744279 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac761636-38a9-4f92-b114-1285fc6a2442-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7\" (UID: \"ac761636-38a9-4f92-b114-1285fc6a2442\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7" Oct 31 03:20:10.755033 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.755003 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmpkl\" (UniqueName: \"kubernetes.io/projected/ac761636-38a9-4f92-b114-1285fc6a2442-kube-api-access-qmpkl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7\" (UID: \"ac761636-38a9-4f92-b114-1285fc6a2442\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7" Oct 31 03:20:10.853604 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.853410 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw"] Oct 31 03:20:10.853787 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.853737 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7" Oct 31 03:20:10.864176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.861470 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:10.872142 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.872113 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 03:20:10.872293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.872268 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw"] Oct 31 03:20:10.872470 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.872454 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 03:20:10.872589 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.872575 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 03:20:10.874048 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.873028 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tw4cw\"" Oct 31 03:20:10.945196 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.945162 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:10.945350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.945208 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gsbc\" (UniqueName: \"kubernetes.io/projected/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-kube-api-access-6gsbc\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:10.945350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.945238 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:10.945350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.945261 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:10.945350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.945294 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:10.945350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.945323 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:10.945520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.945351 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:10.945520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.945439 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:10.945520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.945468 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:10.975474 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:10.974814 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:11.046790 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.046742 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:11.046958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.046798 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gsbc\" (UniqueName: \"kubernetes.io/projected/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-kube-api-access-6gsbc\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:11.046958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.046829 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:11.046958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.046858 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:11.046958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.046893 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:11.046958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.046921 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:11.046958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.046947 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:11.047311 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.047044 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:11.047311 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.047074 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:11.047850 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.047817 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:11.048071 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.048051 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:11.048602 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.048579 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:11.049409 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.049269 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:11.049850 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.049826 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:11.056829 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.056745 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:11.057169 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.057110 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gsbc\" (UniqueName: \"kubernetes.io/projected/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-kube-api-access-6gsbc\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:11.063325 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.063296 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:11.075534 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.075467 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-k9bcw\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:11.108037 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.107931 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:11.179056 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.178969 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7"] Oct 31 03:20:11.197465 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.197424 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:11.237520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.237470 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:11.480950 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.480912 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7" event={"ID":"ac761636-38a9-4f92-b114-1285fc6a2442","Type":"ContainerStarted","Data":"f284b9928ed48e10e32f6778e3c2b7e6c131dc8c7c58bdedb3907c72bb6cfbe2"} Oct 31 03:20:11.508323 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:11.508291 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw"] Oct 31 03:20:11.512476 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:20:11.512445 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e563544_8d00_4f0f_bd8a_c0cfc2c5c666.slice/crio-fafe8613676fc4583983dc2607627e434123b8a96773fcc8a6ff3c0275182239 WatchSource:0}: Error finding container fafe8613676fc4583983dc2607627e434123b8a96773fcc8a6ff3c0275182239: Status 404 returned error can't find the container with id fafe8613676fc4583983dc2607627e434123b8a96773fcc8a6ff3c0275182239 Oct 31 03:20:12.492911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:12.492855 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" event={"ID":"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666","Type":"ContainerStarted","Data":"fafe8613676fc4583983dc2607627e434123b8a96773fcc8a6ff3c0275182239"} Oct 31 03:20:13.605311 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:13.605246 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:15.121638 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:15.121517 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:15.246198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:15.246105 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:17.148448 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:20:17.148396 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:20:19.634989 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:19.634951 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:20:19.635333 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:19.635048 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:20:19.635333 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:19.635092 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:20:20.564488 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:20.563821 5490 generic.go:356] "Generic (PLEG): container finished" podID="a67f04ce-194b-4404-9180-8076e089c10c" containerID="8dbb0dc95096a24bc0eabacfc47fb90aee3907fd54f0a91491d2509208ac57ad" exitCode=0 Oct 31 03:20:20.564488 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:20.563922 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn" event={"ID":"a67f04ce-194b-4404-9180-8076e089c10c","Type":"ContainerDied","Data":"8dbb0dc95096a24bc0eabacfc47fb90aee3907fd54f0a91491d2509208ac57ad"} Oct 31 03:20:20.566273 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:20.566241 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" event={"ID":"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666","Type":"ContainerStarted","Data":"5095b4b424c27e0d1b4508340a2ef081f59d900607606f5547cd9d9696dd72b7"} Oct 31 03:20:20.605117 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:20.605026 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" podStartSLOduration=2.487896642 podStartE2EDuration="10.605010267s" totalImagesPullingTime="8.117113625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:20.603885439 +0000 UTC m=+255.138580394" watchObservedRunningTime="2026-10-31 03:20:20.605010267 +0000 UTC m=+255.139705224" Oct 31 03:20:21.198549 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:21.198516 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:21.204038 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:21.204011 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:21.571458 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:21.571426 5490 generic.go:356] "Generic (PLEG): container finished" podID="18ef0547-2e2f-44ef-87dd-e307a41d6f32" containerID="736ef5cfc2e27e82fde5d9791b4e70bd0e6d5a88f670542081188aa676132d5f" exitCode=0 Oct 31 03:20:21.572182 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:21.572158 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p" event={"ID":"18ef0547-2e2f-44ef-87dd-e307a41d6f32","Type":"ContainerDied","Data":"736ef5cfc2e27e82fde5d9791b4e70bd0e6d5a88f670542081188aa676132d5f"} Oct 31 03:20:21.573095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:21.573068 5490 generic.go:356] "Generic (PLEG): container finished" podID="3b81b232-bcdc-4208-82f7-78e7fd4674d9" containerID="9884c3760df5f08eb2a4a394e5cd99d473bca340945f0cf773a589c9a6f3ed72" exitCode=0 Oct 31 03:20:21.573195 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:21.573098 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm" event={"ID":"3b81b232-bcdc-4208-82f7-78e7fd4674d9","Type":"ContainerDied","Data":"9884c3760df5f08eb2a4a394e5cd99d473bca340945f0cf773a589c9a6f3ed72"} Oct 31 03:20:21.575061 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:21.575040 5490 generic.go:356] "Generic (PLEG): container finished" podID="ac761636-38a9-4f92-b114-1285fc6a2442" containerID="7e765e71756aed33b881ee6a44054316f758ca73ddf846241bfec1c3a1789f79" exitCode=0 Oct 31 03:20:21.575141 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:21.575107 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7" event={"ID":"ac761636-38a9-4f92-b114-1285fc6a2442","Type":"ContainerDied","Data":"7e765e71756aed33b881ee6a44054316f758ca73ddf846241bfec1c3a1789f79"} Oct 31 03:20:21.576707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:21.576666 5490 generic.go:356] "Generic (PLEG): container finished" podID="7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f" containerID="2798522e358a12a3171326c3111d3e368811bec35da82631ecef2344e862220a" exitCode=0 Oct 31 03:20:21.576768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:21.576709 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6" event={"ID":"7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f","Type":"ContainerDied","Data":"2798522e358a12a3171326c3111d3e368811bec35da82631ecef2344e862220a"} Oct 31 03:20:21.578350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:21.578328 5490 generic.go:356] "Generic (PLEG): container finished" podID="8e5abc82-cd8d-4810-911a-fbe67b573695" containerID="194cc0c8692fb0d371a1d8e9d816678ef3be01ff99f18ef35be506c7d72aa9e6" exitCode=0 Oct 31 03:20:21.578446 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:21.578397 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9" event={"ID":"8e5abc82-cd8d-4810-911a-fbe67b573695","Type":"ContainerDied","Data":"194cc0c8692fb0d371a1d8e9d816678ef3be01ff99f18ef35be506c7d72aa9e6"} Oct 31 03:20:21.578778 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:21.578725 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:21.580801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:21.580785 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:20:22.584091 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:22.584059 5490 generic.go:356] "Generic (PLEG): container finished" podID="ac761636-38a9-4f92-b114-1285fc6a2442" containerID="f600e26228c629c93cddd9c749771b0c1bbb9012ac53297a42e57d3915287d04" exitCode=0 Oct 31 03:20:22.584534 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:22.584139 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7" event={"ID":"ac761636-38a9-4f92-b114-1285fc6a2442","Type":"ContainerDied","Data":"f600e26228c629c93cddd9c749771b0c1bbb9012ac53297a42e57d3915287d04"} Oct 31 03:20:22.586240 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:22.586214 5490 generic.go:356] "Generic (PLEG): container finished" podID="5b19c45f-afd6-45eb-9a42-4afff1071032" containerID="01ba7a40856e3b7821b71e1998595c08c09924e62da0e209005a5d94e2064fcd" exitCode=0 Oct 31 03:20:22.586947 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:22.586375 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq" event={"ID":"5b19c45f-afd6-45eb-9a42-4afff1071032","Type":"ContainerDied","Data":"01ba7a40856e3b7821b71e1998595c08c09924e62da0e209005a5d94e2064fcd"} Oct 31 03:20:22.589053 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:22.589029 5490 generic.go:356] "Generic (PLEG): container finished" podID="a67f04ce-194b-4404-9180-8076e089c10c" containerID="30eb100e604d741c84079cbdae68b16ba63155a9dc2417edd24740b87b33ca06" exitCode=0 Oct 31 03:20:22.589173 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:22.589116 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn" event={"ID":"a67f04ce-194b-4404-9180-8076e089c10c","Type":"ContainerDied","Data":"30eb100e604d741c84079cbdae68b16ba63155a9dc2417edd24740b87b33ca06"} Oct 31 03:20:22.591299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:22.591281 5490 generic.go:356] "Generic (PLEG): container finished" podID="18ef0547-2e2f-44ef-87dd-e307a41d6f32" containerID="89dcbd94f3e2dc38aaf7b32f9f197cf4f7b8e94b245fbebaac95fd3254b9388c" exitCode=0 Oct 31 03:20:22.591381 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:22.591362 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p" event={"ID":"18ef0547-2e2f-44ef-87dd-e307a41d6f32","Type":"ContainerDied","Data":"89dcbd94f3e2dc38aaf7b32f9f197cf4f7b8e94b245fbebaac95fd3254b9388c"} Oct 31 03:20:23.599589 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:23.599548 5490 generic.go:356] "Generic (PLEG): container finished" podID="8e5abc82-cd8d-4810-911a-fbe67b573695" containerID="792d5c7da0b902686fdf5967d1c217adff0bc4cf3161b86e7c90625de619064e" exitCode=0 Oct 31 03:20:23.600088 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:23.599726 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9" event={"ID":"8e5abc82-cd8d-4810-911a-fbe67b573695","Type":"ContainerDied","Data":"792d5c7da0b902686fdf5967d1c217adff0bc4cf3161b86e7c90625de619064e"} Oct 31 03:20:23.606544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:23.606514 5490 generic.go:356] "Generic (PLEG): container finished" podID="3b81b232-bcdc-4208-82f7-78e7fd4674d9" containerID="217b8c8accc761626ef11590edf6a2752bc1f94acf8bb995cbb9fc0e8eaf1826" exitCode=0 Oct 31 03:20:23.607328 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:23.606736 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm" event={"ID":"3b81b232-bcdc-4208-82f7-78e7fd4674d9","Type":"ContainerDied","Data":"217b8c8accc761626ef11590edf6a2752bc1f94acf8bb995cbb9fc0e8eaf1826"} Oct 31 03:20:25.626711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:25.626619 5490 generic.go:356] "Generic (PLEG): container finished" podID="7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f" containerID="e429738c16ef595827df8610fa587aadf33cf4c02a17175562eb93ee3e60af03" exitCode=0 Oct 31 03:20:25.626711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:25.626670 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6" event={"ID":"7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f","Type":"ContainerDied","Data":"e429738c16ef595827df8610fa587aadf33cf4c02a17175562eb93ee3e60af03"} Oct 31 03:20:27.651962 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:27.651874 5490 generic.go:356] "Generic (PLEG): container finished" podID="5b19c45f-afd6-45eb-9a42-4afff1071032" containerID="3bdd94db8b2c62ac08f3f06f5d39257c34d89560de161f08b634dbf75bf4fa75" exitCode=0 Oct 31 03:20:27.652412 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:27.651965 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq" event={"ID":"5b19c45f-afd6-45eb-9a42-4afff1071032","Type":"ContainerDied","Data":"3bdd94db8b2c62ac08f3f06f5d39257c34d89560de161f08b634dbf75bf4fa75"} Oct 31 03:20:37.732904 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:37.732823 5490 generic.go:356] "Generic (PLEG): container finished" podID="ac761636-38a9-4f92-b114-1285fc6a2442" containerID="3b086ebf08c86375958448de0f3e00a00cbe82d49fbd207841d150bc3485c9fa" exitCode=0 Oct 31 03:20:37.733432 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:37.733017 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7" event={"ID":"ac761636-38a9-4f92-b114-1285fc6a2442","Type":"ContainerDied","Data":"3b086ebf08c86375958448de0f3e00a00cbe82d49fbd207841d150bc3485c9fa"} Oct 31 03:20:37.736152 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:37.736120 5490 generic.go:356] "Generic (PLEG): container finished" podID="18ef0547-2e2f-44ef-87dd-e307a41d6f32" containerID="941794621f48e57d9880343ffba9797e580f2049261954cfb5a9f481f89237da" exitCode=0 Oct 31 03:20:37.736297 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:37.736175 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p" event={"ID":"18ef0547-2e2f-44ef-87dd-e307a41d6f32","Type":"ContainerDied","Data":"941794621f48e57d9880343ffba9797e580f2049261954cfb5a9f481f89237da"} Oct 31 03:20:38.744270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:38.744133 5490 generic.go:356] "Generic (PLEG): container finished" podID="8e5abc82-cd8d-4810-911a-fbe67b573695" containerID="e3c0dda1d794ef30c658f95af9b8bc65cc6fd5da1cb7023e5958251099205f1e" exitCode=0 Oct 31 03:20:38.744270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:38.744215 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9" event={"ID":"8e5abc82-cd8d-4810-911a-fbe67b573695","Type":"ContainerDied","Data":"e3c0dda1d794ef30c658f95af9b8bc65cc6fd5da1cb7023e5958251099205f1e"} Oct 31 03:20:38.747114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:38.747082 5490 generic.go:356] "Generic (PLEG): container finished" podID="a67f04ce-194b-4404-9180-8076e089c10c" containerID="8f8e407d4d91cbacb4ad86ae30e4c5e5ae57479bdd0646cd09646cbb3eb048e8" exitCode=0 Oct 31 03:20:38.747256 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:38.747194 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn" event={"ID":"a67f04ce-194b-4404-9180-8076e089c10c","Type":"ContainerDied","Data":"8f8e407d4d91cbacb4ad86ae30e4c5e5ae57479bdd0646cd09646cbb3eb048e8"} Oct 31 03:20:38.750052 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:38.750024 5490 generic.go:356] "Generic (PLEG): container finished" podID="3b81b232-bcdc-4208-82f7-78e7fd4674d9" containerID="7ae6140f40e07acb1494ae4b4f7dbeb828674b865b03e315dd6cda15455e058a" exitCode=0 Oct 31 03:20:38.750542 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:38.750520 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm" event={"ID":"3b81b232-bcdc-4208-82f7-78e7fd4674d9","Type":"ContainerDied","Data":"7ae6140f40e07acb1494ae4b4f7dbeb828674b865b03e315dd6cda15455e058a"} Oct 31 03:20:38.890190 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:38.890149 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7" Oct 31 03:20:38.924883 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:38.924849 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p" Oct 31 03:20:38.941982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:38.941941 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac761636-38a9-4f92-b114-1285fc6a2442-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac761636-38a9-4f92-b114-1285fc6a2442-bundle\") pod \"ac761636-38a9-4f92-b114-1285fc6a2442\" (UID: \"ac761636-38a9-4f92-b114-1285fc6a2442\") " Oct 31 03:20:38.942163 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:38.941995 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac761636-38a9-4f92-b114-1285fc6a2442-util\" (UniqueName: \"kubernetes.io/empty-dir/ac761636-38a9-4f92-b114-1285fc6a2442-util\") pod \"ac761636-38a9-4f92-b114-1285fc6a2442\" (UID: \"ac761636-38a9-4f92-b114-1285fc6a2442\") " Oct 31 03:20:38.942163 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:38.942030 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac761636-38a9-4f92-b114-1285fc6a2442-kube-api-access-qmpkl\" (UniqueName: \"kubernetes.io/projected/ac761636-38a9-4f92-b114-1285fc6a2442-kube-api-access-qmpkl\") pod \"ac761636-38a9-4f92-b114-1285fc6a2442\" (UID: \"ac761636-38a9-4f92-b114-1285fc6a2442\") " Oct 31 03:20:38.943145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:38.943112 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac761636-38a9-4f92-b114-1285fc6a2442-bundle" pod "ac761636-38a9-4f92-b114-1285fc6a2442" (UID: "ac761636-38a9-4f92-b114-1285fc6a2442"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:38.948033 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:38.948000 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac761636-38a9-4f92-b114-1285fc6a2442-util" pod "ac761636-38a9-4f92-b114-1285fc6a2442" (UID: "ac761636-38a9-4f92-b114-1285fc6a2442"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:38.950730 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:38.950702 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac761636-38a9-4f92-b114-1285fc6a2442-kube-api-access-qmpkl" pod "ac761636-38a9-4f92-b114-1285fc6a2442" (UID: "ac761636-38a9-4f92-b114-1285fc6a2442"). InnerVolumeSpecName "kube-api-access-qmpkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:39.043064 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.043021 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ef0547-2e2f-44ef-87dd-e307a41d6f32-kube-api-access-rvwbt\" (UniqueName: \"kubernetes.io/projected/18ef0547-2e2f-44ef-87dd-e307a41d6f32-kube-api-access-rvwbt\") pod \"18ef0547-2e2f-44ef-87dd-e307a41d6f32\" (UID: \"18ef0547-2e2f-44ef-87dd-e307a41d6f32\") " Oct 31 03:20:39.043259 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.043140 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ef0547-2e2f-44ef-87dd-e307a41d6f32-bundle\" (UniqueName: \"kubernetes.io/empty-dir/18ef0547-2e2f-44ef-87dd-e307a41d6f32-bundle\") pod \"18ef0547-2e2f-44ef-87dd-e307a41d6f32\" (UID: \"18ef0547-2e2f-44ef-87dd-e307a41d6f32\") " Oct 31 03:20:39.043259 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.043183 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ef0547-2e2f-44ef-87dd-e307a41d6f32-util\" (UniqueName: \"kubernetes.io/empty-dir/18ef0547-2e2f-44ef-87dd-e307a41d6f32-util\") pod \"18ef0547-2e2f-44ef-87dd-e307a41d6f32\" (UID: \"18ef0547-2e2f-44ef-87dd-e307a41d6f32\") " Oct 31 03:20:39.043392 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.043382 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmpkl\" (UniqueName: \"kubernetes.io/projected/ac761636-38a9-4f92-b114-1285fc6a2442-kube-api-access-qmpkl\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:39.043449 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.043397 5490 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac761636-38a9-4f92-b114-1285fc6a2442-bundle\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:39.043449 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.043407 5490 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac761636-38a9-4f92-b114-1285fc6a2442-util\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:39.044238 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.044200 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ef0547-2e2f-44ef-87dd-e307a41d6f32-bundle" pod "18ef0547-2e2f-44ef-87dd-e307a41d6f32" (UID: "18ef0547-2e2f-44ef-87dd-e307a41d6f32"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:39.048004 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.047970 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ef0547-2e2f-44ef-87dd-e307a41d6f32-kube-api-access-rvwbt" pod "18ef0547-2e2f-44ef-87dd-e307a41d6f32" (UID: "18ef0547-2e2f-44ef-87dd-e307a41d6f32"). InnerVolumeSpecName "kube-api-access-rvwbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:39.050288 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.050263 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ef0547-2e2f-44ef-87dd-e307a41d6f32-util" pod "18ef0547-2e2f-44ef-87dd-e307a41d6f32" (UID: "18ef0547-2e2f-44ef-87dd-e307a41d6f32"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:39.144777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.144666 5490 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/18ef0547-2e2f-44ef-87dd-e307a41d6f32-bundle\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:39.144777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.144723 5490 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/18ef0547-2e2f-44ef-87dd-e307a41d6f32-util\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:39.144777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.144738 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvwbt\" (UniqueName: \"kubernetes.io/projected/18ef0547-2e2f-44ef-87dd-e307a41d6f32-kube-api-access-rvwbt\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:39.756365 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.756323 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7" event={"ID":"ac761636-38a9-4f92-b114-1285fc6a2442","Type":"ContainerDied","Data":"f284b9928ed48e10e32f6778e3c2b7e6c131dc8c7c58bdedb3907c72bb6cfbe2"} Oct 31 03:20:39.756365 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.756366 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719mqj7" Oct 31 03:20:39.756942 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.756387 5490 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f284b9928ed48e10e32f6778e3c2b7e6c131dc8c7c58bdedb3907c72bb6cfbe2" Oct 31 03:20:39.759398 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.759371 5490 generic.go:356] "Generic (PLEG): container finished" podID="7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f" containerID="d5acc685421806690275eba1372917cb1ffdc06ceff289865bb8e3b116e8134f" exitCode=0 Oct 31 03:20:39.759538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.759454 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6" event={"ID":"7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f","Type":"ContainerDied","Data":"d5acc685421806690275eba1372917cb1ffdc06ceff289865bb8e3b116e8134f"} Oct 31 03:20:39.761630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.761611 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p" Oct 31 03:20:39.761745 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.761651 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9ss2p" event={"ID":"18ef0547-2e2f-44ef-87dd-e307a41d6f32","Type":"ContainerDied","Data":"a536d33c6b15ba832f8b5242b69e25406b2ab44ef4db28c27f4c322b64fad17d"} Oct 31 03:20:39.761745 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.761714 5490 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a536d33c6b15ba832f8b5242b69e25406b2ab44ef4db28c27f4c322b64fad17d" Oct 31 03:20:39.924476 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.924446 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:39.926540 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.926516 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn" Oct 31 03:20:39.929759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.929735 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm" Oct 31 03:20:39.957508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:39.957481 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9" Oct 31 03:20:40.052593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.052508 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67f04ce-194b-4404-9180-8076e089c10c-util\" (UniqueName: \"kubernetes.io/empty-dir/a67f04ce-194b-4404-9180-8076e089c10c-util\") pod \"a67f04ce-194b-4404-9180-8076e089c10c\" (UID: \"a67f04ce-194b-4404-9180-8076e089c10c\") " Oct 31 03:20:40.052593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.052551 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b81b232-bcdc-4208-82f7-78e7fd4674d9-kube-api-access-mfxk9\" (UniqueName: \"kubernetes.io/projected/3b81b232-bcdc-4208-82f7-78e7fd4674d9-kube-api-access-mfxk9\") pod \"3b81b232-bcdc-4208-82f7-78e7fd4674d9\" (UID: \"3b81b232-bcdc-4208-82f7-78e7fd4674d9\") " Oct 31 03:20:40.052894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.052607 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e5abc82-cd8d-4810-911a-fbe67b573695-util\" (UniqueName: \"kubernetes.io/empty-dir/8e5abc82-cd8d-4810-911a-fbe67b573695-util\") pod \"8e5abc82-cd8d-4810-911a-fbe67b573695\" (UID: \"8e5abc82-cd8d-4810-911a-fbe67b573695\") " Oct 31 03:20:40.052894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.052716 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e5abc82-cd8d-4810-911a-fbe67b573695-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e5abc82-cd8d-4810-911a-fbe67b573695-bundle\") pod \"8e5abc82-cd8d-4810-911a-fbe67b573695\" (UID: \"8e5abc82-cd8d-4810-911a-fbe67b573695\") " Oct 31 03:20:40.052894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.052751 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b81b232-bcdc-4208-82f7-78e7fd4674d9-util\" (UniqueName: \"kubernetes.io/empty-dir/3b81b232-bcdc-4208-82f7-78e7fd4674d9-util\") pod \"3b81b232-bcdc-4208-82f7-78e7fd4674d9\" (UID: \"3b81b232-bcdc-4208-82f7-78e7fd4674d9\") " Oct 31 03:20:40.052894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.052810 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e5abc82-cd8d-4810-911a-fbe67b573695-kube-api-access-msjr6\" (UniqueName: \"kubernetes.io/projected/8e5abc82-cd8d-4810-911a-fbe67b573695-kube-api-access-msjr6\") pod \"8e5abc82-cd8d-4810-911a-fbe67b573695\" (UID: \"8e5abc82-cd8d-4810-911a-fbe67b573695\") " Oct 31 03:20:40.052894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.052838 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a67f04ce-194b-4404-9180-8076e089c10c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a67f04ce-194b-4404-9180-8076e089c10c-bundle\") pod \"a67f04ce-194b-4404-9180-8076e089c10c\" (UID: \"a67f04ce-194b-4404-9180-8076e089c10c\") " Oct 31 03:20:40.052894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.052883 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b81b232-bcdc-4208-82f7-78e7fd4674d9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b81b232-bcdc-4208-82f7-78e7fd4674d9-bundle\") pod \"3b81b232-bcdc-4208-82f7-78e7fd4674d9\" (UID: \"3b81b232-bcdc-4208-82f7-78e7fd4674d9\") " Oct 31 03:20:40.053205 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.052918 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67f04ce-194b-4404-9180-8076e089c10c-kube-api-access-88ggk\" (UniqueName: \"kubernetes.io/projected/a67f04ce-194b-4404-9180-8076e089c10c-kube-api-access-88ggk\") pod \"a67f04ce-194b-4404-9180-8076e089c10c\" (UID: \"a67f04ce-194b-4404-9180-8076e089c10c\") " Oct 31 03:20:40.054002 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.053966 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5abc82-cd8d-4810-911a-fbe67b573695-bundle" pod "8e5abc82-cd8d-4810-911a-fbe67b573695" (UID: "8e5abc82-cd8d-4810-911a-fbe67b573695"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:40.054994 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.054957 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67f04ce-194b-4404-9180-8076e089c10c-bundle" pod "a67f04ce-194b-4404-9180-8076e089c10c" (UID: "a67f04ce-194b-4404-9180-8076e089c10c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:40.055103 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.055071 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b81b232-bcdc-4208-82f7-78e7fd4674d9-bundle" pod "3b81b232-bcdc-4208-82f7-78e7fd4674d9" (UID: "3b81b232-bcdc-4208-82f7-78e7fd4674d9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:40.055648 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.055622 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67f04ce-194b-4404-9180-8076e089c10c-kube-api-access-88ggk" pod "a67f04ce-194b-4404-9180-8076e089c10c" (UID: "a67f04ce-194b-4404-9180-8076e089c10c"). InnerVolumeSpecName "kube-api-access-88ggk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:40.055759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.055696 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b81b232-bcdc-4208-82f7-78e7fd4674d9-kube-api-access-mfxk9" pod "3b81b232-bcdc-4208-82f7-78e7fd4674d9" (UID: "3b81b232-bcdc-4208-82f7-78e7fd4674d9"). InnerVolumeSpecName "kube-api-access-mfxk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:40.055819 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.055761 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5abc82-cd8d-4810-911a-fbe67b573695-kube-api-access-msjr6" pod "8e5abc82-cd8d-4810-911a-fbe67b573695" (UID: "8e5abc82-cd8d-4810-911a-fbe67b573695"). InnerVolumeSpecName "kube-api-access-msjr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:40.060018 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.059989 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b81b232-bcdc-4208-82f7-78e7fd4674d9-util" pod "3b81b232-bcdc-4208-82f7-78e7fd4674d9" (UID: "3b81b232-bcdc-4208-82f7-78e7fd4674d9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:40.060156 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.060124 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5abc82-cd8d-4810-911a-fbe67b573695-util" pod "8e5abc82-cd8d-4810-911a-fbe67b573695" (UID: "8e5abc82-cd8d-4810-911a-fbe67b573695"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:40.062373 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.062350 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a67f04ce-194b-4404-9180-8076e089c10c-util" pod "a67f04ce-194b-4404-9180-8076e089c10c" (UID: "a67f04ce-194b-4404-9180-8076e089c10c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:40.154127 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.154087 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msjr6\" (UniqueName: \"kubernetes.io/projected/8e5abc82-cd8d-4810-911a-fbe67b573695-kube-api-access-msjr6\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:40.154127 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.154120 5490 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a67f04ce-194b-4404-9180-8076e089c10c-bundle\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:40.154127 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.154135 5490 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b81b232-bcdc-4208-82f7-78e7fd4674d9-bundle\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:40.154402 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.154149 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88ggk\" (UniqueName: \"kubernetes.io/projected/a67f04ce-194b-4404-9180-8076e089c10c-kube-api-access-88ggk\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:40.154402 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.154162 5490 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a67f04ce-194b-4404-9180-8076e089c10c-util\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:40.154402 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.154177 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfxk9\" (UniqueName: \"kubernetes.io/projected/3b81b232-bcdc-4208-82f7-78e7fd4674d9-kube-api-access-mfxk9\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:40.154402 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.154188 5490 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e5abc82-cd8d-4810-911a-fbe67b573695-util\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:40.154402 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.154199 5490 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e5abc82-cd8d-4810-911a-fbe67b573695-bundle\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:40.154402 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.154212 5490 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b81b232-bcdc-4208-82f7-78e7fd4674d9-util\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:40.766953 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.766917 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9" event={"ID":"8e5abc82-cd8d-4810-911a-fbe67b573695","Type":"ContainerDied","Data":"f11ac91718699b6da0f4e1b76ea587b9114a5b0b89d6f4f86301546d5a794334"} Oct 31 03:20:40.767410 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.766976 5490 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f11ac91718699b6da0f4e1b76ea587b9114a5b0b89d6f4f86301546d5a794334" Oct 31 03:20:40.767410 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.766928 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bcsz9" Oct 31 03:20:40.769110 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.769083 5490 generic.go:356] "Generic (PLEG): container finished" podID="5b19c45f-afd6-45eb-9a42-4afff1071032" containerID="99ac5bb4557118161d871e0f394c40ac388bfce3694ebfb8c5d30faba9cd1c3c" exitCode=0 Oct 31 03:20:40.769229 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.769120 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq" event={"ID":"5b19c45f-afd6-45eb-9a42-4afff1071032","Type":"ContainerDied","Data":"99ac5bb4557118161d871e0f394c40ac388bfce3694ebfb8c5d30faba9cd1c3c"} Oct 31 03:20:40.771052 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.771036 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn" Oct 31 03:20:40.771163 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.771052 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847am2lcn" event={"ID":"a67f04ce-194b-4404-9180-8076e089c10c","Type":"ContainerDied","Data":"5024a1b7cc35f5037c581efcd5f34902e5a88d94beedecb4e32d2b4d1095938c"} Oct 31 03:20:40.771163 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.771087 5490 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5024a1b7cc35f5037c581efcd5f34902e5a88d94beedecb4e32d2b4d1095938c" Oct 31 03:20:40.773045 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.773028 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm" Oct 31 03:20:40.773148 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.773060 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68vjnm" event={"ID":"3b81b232-bcdc-4208-82f7-78e7fd4674d9","Type":"ContainerDied","Data":"3fbb31aac9b2c6508abe921c713bfaff06ec5ece6b9bb4c83bbdf495cd80158c"} Oct 31 03:20:40.773148 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.773096 5490 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fbb31aac9b2c6508abe921c713bfaff06ec5ece6b9bb4c83bbdf495cd80158c" Oct 31 03:20:40.894788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.894762 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6" Oct 31 03:20:40.961928 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.961887 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-kube-api-access-j4lt9\" (UniqueName: \"kubernetes.io/projected/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-kube-api-access-j4lt9\") pod \"7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f\" (UID: \"7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f\") " Oct 31 03:20:40.962131 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.961973 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-util\" (UniqueName: \"kubernetes.io/empty-dir/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-util\") pod \"7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f\" (UID: \"7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f\") " Oct 31 03:20:40.962131 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.962013 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-bundle\") pod \"7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f\" (UID: \"7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f\") " Oct 31 03:20:40.962619 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.962593 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-bundle" pod "7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f" (UID: "7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:40.964112 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.964091 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-kube-api-access-j4lt9" pod "7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f" (UID: "7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f"). InnerVolumeSpecName "kube-api-access-j4lt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:40.967028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:40.967006 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-util" pod "7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f" (UID: "7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:41.063101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:41.063017 5490 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-util\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:41.063101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:41.063043 5490 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-bundle\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:41.063101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:41.063053 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4lt9\" (UniqueName: \"kubernetes.io/projected/7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f-kube-api-access-j4lt9\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:41.778915 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:41.778880 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6" Oct 31 03:20:41.779299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:41.778888 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cv4zb6" event={"ID":"7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f","Type":"ContainerDied","Data":"e2b4614d35dce556327a9642c1c6d8c59244d1e2753f4f8004a15fcac9acd81d"} Oct 31 03:20:41.779299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:41.779033 5490 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2b4614d35dce556327a9642c1c6d8c59244d1e2753f4f8004a15fcac9acd81d" Oct 31 03:20:41.901263 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:41.901237 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq" Oct 31 03:20:41.970663 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:41.970623 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b19c45f-afd6-45eb-9a42-4afff1071032-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b19c45f-afd6-45eb-9a42-4afff1071032-bundle\") pod \"5b19c45f-afd6-45eb-9a42-4afff1071032\" (UID: \"5b19c45f-afd6-45eb-9a42-4afff1071032\") " Oct 31 03:20:41.970855 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:41.970700 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b19c45f-afd6-45eb-9a42-4afff1071032-util\" (UniqueName: \"kubernetes.io/empty-dir/5b19c45f-afd6-45eb-9a42-4afff1071032-util\") pod \"5b19c45f-afd6-45eb-9a42-4afff1071032\" (UID: \"5b19c45f-afd6-45eb-9a42-4afff1071032\") " Oct 31 03:20:41.970855 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:41.970800 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b19c45f-afd6-45eb-9a42-4afff1071032-kube-api-access-pfnff\" (UniqueName: \"kubernetes.io/projected/5b19c45f-afd6-45eb-9a42-4afff1071032-kube-api-access-pfnff\") pod \"5b19c45f-afd6-45eb-9a42-4afff1071032\" (UID: \"5b19c45f-afd6-45eb-9a42-4afff1071032\") " Oct 31 03:20:41.971082 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:41.971057 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b19c45f-afd6-45eb-9a42-4afff1071032-bundle" pod "5b19c45f-afd6-45eb-9a42-4afff1071032" (UID: "5b19c45f-afd6-45eb-9a42-4afff1071032"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:41.972850 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:41.972828 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b19c45f-afd6-45eb-9a42-4afff1071032-kube-api-access-pfnff" pod "5b19c45f-afd6-45eb-9a42-4afff1071032" (UID: "5b19c45f-afd6-45eb-9a42-4afff1071032"). InnerVolumeSpecName "kube-api-access-pfnff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:20:41.976269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:41.976247 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b19c45f-afd6-45eb-9a42-4afff1071032-util" pod "5b19c45f-afd6-45eb-9a42-4afff1071032" (UID: "5b19c45f-afd6-45eb-9a42-4afff1071032"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:20:42.071730 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:42.071668 5490 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5b19c45f-afd6-45eb-9a42-4afff1071032-util\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:42.071730 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:42.071726 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfnff\" (UniqueName: \"kubernetes.io/projected/5b19c45f-afd6-45eb-9a42-4afff1071032-kube-api-access-pfnff\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:42.071730 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:42.071737 5490 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5b19c45f-afd6-45eb-9a42-4afff1071032-bundle\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:20:42.315118 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:42.315077 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:42.449030 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:42.448991 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:42.771995 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:42.771962 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:42.785918 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:42.785875 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq" event={"ID":"5b19c45f-afd6-45eb-9a42-4afff1071032","Type":"ContainerDied","Data":"48db54c3d3b6ba63824c8bc32e66344e81920ad3b1777bdfdbfc794331934008"} Oct 31 03:20:42.785918 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:42.785914 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ggmkq" Oct 31 03:20:42.786377 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:42.785944 5490 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48db54c3d3b6ba63824c8bc32e66344e81920ad3b1777bdfdbfc794331934008" Oct 31 03:20:42.917838 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:42.917801 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:43.228460 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.228423 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:43.302728 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.302656 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl"] Oct 31 03:20:43.303092 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303075 5490 cpu_manager.go:414] "containerMap: removing container" podUID="a67f04ce-194b-4404-9180-8076e089c10c" containerName="util" Oct 31 03:20:43.303180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303094 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67f04ce-194b-4404-9180-8076e089c10c" containerName="util" Oct 31 03:20:43.303180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303108 5490 cpu_manager.go:414] "containerMap: removing container" podUID="8e5abc82-cd8d-4810-911a-fbe67b573695" containerName="util" Oct 31 03:20:43.303180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303115 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5abc82-cd8d-4810-911a-fbe67b573695" containerName="util" Oct 31 03:20:43.303180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303129 5490 cpu_manager.go:414] "containerMap: removing container" podUID="18ef0547-2e2f-44ef-87dd-e307a41d6f32" containerName="pull" Oct 31 03:20:43.303180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303136 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ef0547-2e2f-44ef-87dd-e307a41d6f32" containerName="pull" Oct 31 03:20:43.303180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303147 5490 cpu_manager.go:414] "containerMap: removing container" podUID="a67f04ce-194b-4404-9180-8076e089c10c" containerName="extract" Oct 31 03:20:43.303180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303153 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67f04ce-194b-4404-9180-8076e089c10c" containerName="extract" Oct 31 03:20:43.303180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303167 5490 cpu_manager.go:414] "containerMap: removing container" podUID="a67f04ce-194b-4404-9180-8076e089c10c" containerName="pull" Oct 31 03:20:43.303180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303174 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67f04ce-194b-4404-9180-8076e089c10c" containerName="pull" Oct 31 03:20:43.303180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303183 5490 cpu_manager.go:414] "containerMap: removing container" podUID="5b19c45f-afd6-45eb-9a42-4afff1071032" containerName="pull" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303190 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b19c45f-afd6-45eb-9a42-4afff1071032" containerName="pull" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303202 5490 cpu_manager.go:414] "containerMap: removing container" podUID="ac761636-38a9-4f92-b114-1285fc6a2442" containerName="util" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303208 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac761636-38a9-4f92-b114-1285fc6a2442" containerName="util" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303217 5490 cpu_manager.go:414] "containerMap: removing container" podUID="3b81b232-bcdc-4208-82f7-78e7fd4674d9" containerName="util" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303223 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b81b232-bcdc-4208-82f7-78e7fd4674d9" containerName="util" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303234 5490 cpu_manager.go:414] "containerMap: removing container" podUID="7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f" containerName="util" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303240 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f" containerName="util" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303254 5490 cpu_manager.go:414] "containerMap: removing container" podUID="5b19c45f-afd6-45eb-9a42-4afff1071032" containerName="util" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303260 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b19c45f-afd6-45eb-9a42-4afff1071032" containerName="util" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303269 5490 cpu_manager.go:414] "containerMap: removing container" podUID="18ef0547-2e2f-44ef-87dd-e307a41d6f32" containerName="extract" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303275 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ef0547-2e2f-44ef-87dd-e307a41d6f32" containerName="extract" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303285 5490 cpu_manager.go:414] "containerMap: removing container" podUID="3b81b232-bcdc-4208-82f7-78e7fd4674d9" containerName="extract" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303291 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b81b232-bcdc-4208-82f7-78e7fd4674d9" containerName="extract" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303304 5490 cpu_manager.go:414] "containerMap: removing container" podUID="18ef0547-2e2f-44ef-87dd-e307a41d6f32" containerName="util" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303311 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ef0547-2e2f-44ef-87dd-e307a41d6f32" containerName="util" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303318 5490 cpu_manager.go:414] "containerMap: removing container" podUID="7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f" containerName="pull" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303323 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f" containerName="pull" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303330 5490 cpu_manager.go:414] "containerMap: removing container" podUID="8e5abc82-cd8d-4810-911a-fbe67b573695" containerName="pull" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303335 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5abc82-cd8d-4810-911a-fbe67b573695" containerName="pull" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303341 5490 cpu_manager.go:414] "containerMap: removing container" podUID="ac761636-38a9-4f92-b114-1285fc6a2442" containerName="pull" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303345 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac761636-38a9-4f92-b114-1285fc6a2442" containerName="pull" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303351 5490 cpu_manager.go:414] "containerMap: removing container" podUID="8e5abc82-cd8d-4810-911a-fbe67b573695" containerName="extract" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303355 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5abc82-cd8d-4810-911a-fbe67b573695" containerName="extract" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303360 5490 cpu_manager.go:414] "containerMap: removing container" podUID="7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f" containerName="extract" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303364 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f" containerName="extract" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303373 5490 cpu_manager.go:414] "containerMap: removing container" podUID="5b19c45f-afd6-45eb-9a42-4afff1071032" containerName="extract" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303377 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b19c45f-afd6-45eb-9a42-4afff1071032" containerName="extract" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303383 5490 cpu_manager.go:414] "containerMap: removing container" podUID="3b81b232-bcdc-4208-82f7-78e7fd4674d9" containerName="pull" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303387 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b81b232-bcdc-4208-82f7-78e7fd4674d9" containerName="pull" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303393 5490 cpu_manager.go:414] "containerMap: removing container" podUID="ac761636-38a9-4f92-b114-1285fc6a2442" containerName="extract" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303398 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac761636-38a9-4f92-b114-1285fc6a2442" containerName="extract" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303442 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b19c45f-afd6-45eb-9a42-4afff1071032" containerName="extract" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303450 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac761636-38a9-4f92-b114-1285fc6a2442" containerName="extract" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303458 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67f04ce-194b-4404-9180-8076e089c10c" containerName="extract" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303466 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ef0547-2e2f-44ef-87dd-e307a41d6f32" containerName="extract" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303471 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b81b232-bcdc-4208-82f7-78e7fd4674d9" containerName="extract" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303477 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e5abc82-cd8d-4810-911a-fbe67b573695" containerName="extract" Oct 31 03:20:43.303627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.303482 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c48eb5a-93d1-4451-b42b-f7bbca7f3e2f" containerName="extract" Oct 31 03:20:43.306671 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.306656 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:43.320227 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.320207 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 03:20:43.332001 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.331958 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 03:20:43.332001 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.331976 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 03:20:43.333538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.333508 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl"] Oct 31 03:20:43.381612 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.381582 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fd89b915-fe3a-4a8a-aaf5-6927987b2e0d-manifests\") pod \"opendatahub-operator-controller-manager-55975df5cc-vtdtl\" (UID: \"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:43.381815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.381623 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fd89b915-fe3a-4a8a-aaf5-6927987b2e0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-vtdtl\" (UID: \"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:43.381815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.381648 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlc2d\" (UniqueName: \"kubernetes.io/projected/fd89b915-fe3a-4a8a-aaf5-6927987b2e0d-kube-api-access-jlc2d\") pod \"opendatahub-operator-controller-manager-55975df5cc-vtdtl\" (UID: \"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:43.381815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.381727 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd89b915-fe3a-4a8a-aaf5-6927987b2e0d-tmp\") pod \"opendatahub-operator-controller-manager-55975df5cc-vtdtl\" (UID: \"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:43.381815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.381768 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd89b915-fe3a-4a8a-aaf5-6927987b2e0d-metrics-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-vtdtl\" (UID: \"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:43.381815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.381815 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd89b915-fe3a-4a8a-aaf5-6927987b2e0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-vtdtl\" (UID: \"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:43.429937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.429899 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-dw2ct\"" Oct 31 03:20:43.429937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.429899 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 03:20:43.430258 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.430235 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 03:20:43.482352 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.482254 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd89b915-fe3a-4a8a-aaf5-6927987b2e0d-metrics-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-vtdtl\" (UID: \"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:43.482352 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.482306 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd89b915-fe3a-4a8a-aaf5-6927987b2e0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-vtdtl\" (UID: \"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:43.482601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.482358 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fd89b915-fe3a-4a8a-aaf5-6927987b2e0d-manifests\") pod \"opendatahub-operator-controller-manager-55975df5cc-vtdtl\" (UID: \"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:43.482601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.482397 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fd89b915-fe3a-4a8a-aaf5-6927987b2e0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-vtdtl\" (UID: \"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:43.482601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.482426 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlc2d\" (UniqueName: \"kubernetes.io/projected/fd89b915-fe3a-4a8a-aaf5-6927987b2e0d-kube-api-access-jlc2d\") pod \"opendatahub-operator-controller-manager-55975df5cc-vtdtl\" (UID: \"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:43.482601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.482457 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd89b915-fe3a-4a8a-aaf5-6927987b2e0d-tmp\") pod \"opendatahub-operator-controller-manager-55975df5cc-vtdtl\" (UID: \"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:43.482850 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.482773 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fd89b915-fe3a-4a8a-aaf5-6927987b2e0d-manifests\") pod \"opendatahub-operator-controller-manager-55975df5cc-vtdtl\" (UID: \"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:43.484857 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.484829 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd89b915-fe3a-4a8a-aaf5-6927987b2e0d-tmp\") pod \"opendatahub-operator-controller-manager-55975df5cc-vtdtl\" (UID: \"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:43.484971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.484915 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fd89b915-fe3a-4a8a-aaf5-6927987b2e0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-vtdtl\" (UID: \"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:43.485038 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.485020 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd89b915-fe3a-4a8a-aaf5-6927987b2e0d-metrics-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-vtdtl\" (UID: \"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:43.485077 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.485054 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fd89b915-fe3a-4a8a-aaf5-6927987b2e0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-vtdtl\" (UID: \"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:43.491397 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.491377 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlc2d\" (UniqueName: \"kubernetes.io/projected/fd89b915-fe3a-4a8a-aaf5-6927987b2e0d-kube-api-access-jlc2d\") pod \"opendatahub-operator-controller-manager-55975df5cc-vtdtl\" (UID: \"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:43.511479 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.511442 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:43.621167 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.621127 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:43.650269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.650228 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:43.804225 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.804138 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:43.922716 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:43.922670 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl"] Oct 31 03:20:43.925325 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:20:43.925299 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd89b915_fe3a_4a8a_aaf5_6927987b2e0d.slice/crio-d36aa14f17896e8128e9bfeedca67ea930021890c65f8c4dbdf2b1a9eea8c7a3 WatchSource:0}: Error finding container d36aa14f17896e8128e9bfeedca67ea930021890c65f8c4dbdf2b1a9eea8c7a3: Status 404 returned error can't find the container with id d36aa14f17896e8128e9bfeedca67ea930021890c65f8c4dbdf2b1a9eea8c7a3 Oct 31 03:20:44.795326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:44.795284 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" event={"ID":"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d","Type":"ContainerStarted","Data":"d36aa14f17896e8128e9bfeedca67ea930021890c65f8c4dbdf2b1a9eea8c7a3"} Oct 31 03:20:45.937126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:45.937034 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:46.534127 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:46.534071 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:47.131248 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:20:47.131212 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:20:47.362031 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:47.361929 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:47.810374 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:47.810329 5490 generic.go:356] "Generic (PLEG): container finished" podID="fd89b915-fe3a-4a8a-aaf5-6927987b2e0d" containerID="47fd0dc08f2b522398707d3c32936a5d70bdd5eee2bc74a0d0dd0ad018daa0ea" exitCode=0 Oct 31 03:20:47.810549 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:47.810401 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" event={"ID":"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d","Type":"ContainerDied","Data":"47fd0dc08f2b522398707d3c32936a5d70bdd5eee2bc74a0d0dd0ad018daa0ea"} Oct 31 03:20:48.816660 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:48.816621 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" event={"ID":"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d","Type":"ContainerStarted","Data":"6e5fed394fb4796510344e30f0eec59f59015e76d311e05c4f6df358cbfd1e25"} Oct 31 03:20:48.817083 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:48.816733 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:20:48.856327 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:48.856271 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" podStartSLOduration=1.9724317 podStartE2EDuration="5.856257675s" totalImagesPullingTime="3.462910228s" totalInitContainerRuntime="420.915747ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:48.854692055 +0000 UTC m=+283.389387010" watchObservedRunningTime="2026-10-31 03:20:48.856257675 +0000 UTC m=+283.390952630" Oct 31 03:20:49.735785 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:49.735756 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl"] Oct 31 03:20:49.739058 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:49.739042 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl" Oct 31 03:20:49.757667 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:49.757642 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 31 03:20:49.757861 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:49.757844 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 31 03:20:49.778595 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:49.778560 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-p6h8f\"" Oct 31 03:20:49.779759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:49.779739 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl"] Oct 31 03:20:49.840288 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:49.840254 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b50aa305-9fe9-48de-a7d1-e8654c325be4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl\" (UID: \"b50aa305-9fe9-48de-a7d1-e8654c325be4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl" Oct 31 03:20:49.840692 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:49.840294 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pbgx\" (UniqueName: \"kubernetes.io/projected/b50aa305-9fe9-48de-a7d1-e8654c325be4-kube-api-access-6pbgx\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl\" (UID: \"b50aa305-9fe9-48de-a7d1-e8654c325be4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl" Oct 31 03:20:49.941208 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:49.941159 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pbgx\" (UniqueName: \"kubernetes.io/projected/b50aa305-9fe9-48de-a7d1-e8654c325be4-kube-api-access-6pbgx\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl\" (UID: \"b50aa305-9fe9-48de-a7d1-e8654c325be4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl" Oct 31 03:20:49.941367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:49.941330 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b50aa305-9fe9-48de-a7d1-e8654c325be4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl\" (UID: \"b50aa305-9fe9-48de-a7d1-e8654c325be4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl" Oct 31 03:20:49.941732 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:49.941708 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b50aa305-9fe9-48de-a7d1-e8654c325be4-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl\" (UID: \"b50aa305-9fe9-48de-a7d1-e8654c325be4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl" Oct 31 03:20:49.954704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:49.954664 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pbgx\" (UniqueName: \"kubernetes.io/projected/b50aa305-9fe9-48de-a7d1-e8654c325be4-kube-api-access-6pbgx\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl\" (UID: \"b50aa305-9fe9-48de-a7d1-e8654c325be4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl" Oct 31 03:20:50.049649 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:50.049563 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl" Oct 31 03:20:50.331788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:50.331710 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl"] Oct 31 03:20:50.336054 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:20:50.336026 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50aa305_9fe9_48de_a7d1_e8654c325be4.slice/crio-ead2b728678d593b4f276538ff0cb9fe328e7cab2f00b3e0a75bd83a7eaf8f91 WatchSource:0}: Error finding container ead2b728678d593b4f276538ff0cb9fe328e7cab2f00b3e0a75bd83a7eaf8f91: Status 404 returned error can't find the container with id ead2b728678d593b4f276538ff0cb9fe328e7cab2f00b3e0a75bd83a7eaf8f91 Oct 31 03:20:50.827840 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:50.827803 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl" event={"ID":"b50aa305-9fe9-48de-a7d1-e8654c325be4","Type":"ContainerStarted","Data":"ead2b728678d593b4f276538ff0cb9fe328e7cab2f00b3e0a75bd83a7eaf8f91"} Oct 31 03:20:51.972853 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:51.972816 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:52.129488 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:52.129441 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:52.839365 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:52.839333 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl" event={"ID":"b50aa305-9fe9-48de-a7d1-e8654c325be4","Type":"ContainerStarted","Data":"a9d66bad762aff4f468dd506bc62ff06d20c24d6e456af461e802e3210830e26"} Oct 31 03:20:52.864342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:52.864288 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl" podStartSLOduration=1.843846162 podStartE2EDuration="3.864273977s" totalImagesPullingTime="2.020427815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:52.862690156 +0000 UTC m=+287.397385128" watchObservedRunningTime="2026-10-31 03:20:52.864273977 +0000 UTC m=+287.398968931" Oct 31 03:20:52.925223 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:52.925160 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:53.493843 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:53.493799 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:54.191861 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:54.191825 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:54.356361 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:54.356326 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:56.368937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:56.368905 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8"] Oct 31 03:20:56.372262 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:56.372244 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" Oct 31 03:20:56.376490 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:56.376461 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 03:20:56.376490 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:56.376489 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 03:20:56.377795 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:56.377776 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-hqnv5\"" Oct 31 03:20:56.381332 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:56.381308 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8"] Oct 31 03:20:56.397903 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:56.397864 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:56.492878 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:56.492813 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tss7b\" (UniqueName: \"kubernetes.io/projected/efd854cd-40a6-4f9f-bd19-cd5d19512dba-kube-api-access-tss7b\") pod \"limitador-operator-controller-manager-5778bb4449-f2tk8\" (UID: \"efd854cd-40a6-4f9f-bd19-cd5d19512dba\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" Oct 31 03:20:56.593639 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:56.593599 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tss7b\" (UniqueName: \"kubernetes.io/projected/efd854cd-40a6-4f9f-bd19-cd5d19512dba-kube-api-access-tss7b\") pod \"limitador-operator-controller-manager-5778bb4449-f2tk8\" (UID: \"efd854cd-40a6-4f9f-bd19-cd5d19512dba\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" Oct 31 03:20:56.605233 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:56.605196 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tss7b\" (UniqueName: \"kubernetes.io/projected/efd854cd-40a6-4f9f-bd19-cd5d19512dba-kube-api-access-tss7b\") pod \"limitador-operator-controller-manager-5778bb4449-f2tk8\" (UID: \"efd854cd-40a6-4f9f-bd19-cd5d19512dba\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" Oct 31 03:20:56.685401 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:56.685359 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" Oct 31 03:20:56.735259 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:56.735224 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:56.890251 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:56.890198 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:56.968898 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:56.968822 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8"] Oct 31 03:20:56.972024 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:20:56.971994 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd854cd_40a6_4f9f_bd19_cd5d19512dba.slice/crio-06ab7405f52169e7bba5402b5d51a638c070b692a8fface1d507dbf85dff239e WatchSource:0}: Error finding container 06ab7405f52169e7bba5402b5d51a638c070b692a8fface1d507dbf85dff239e: Status 404 returned error can't find the container with id 06ab7405f52169e7bba5402b5d51a638c070b692a8fface1d507dbf85dff239e Oct 31 03:20:57.209604 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:57.209569 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-h5bqj"] Oct 31 03:20:57.214265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:57.214246 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h5bqj" Oct 31 03:20:57.222168 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:57.222132 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 03:20:57.222295 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:57.222270 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-bxmm6\"" Oct 31 03:20:57.222357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:57.222313 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 03:20:57.222981 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:57.222957 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-h5bqj"] Oct 31 03:20:57.301570 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:57.301530 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc99x\" (UniqueName: \"kubernetes.io/projected/6504ab4d-1fc3-4116-838a-e3708400931b-kube-api-access-xc99x\") pod \"cert-manager-cainjector-5c57bbd4fc-h5bqj\" (UID: \"6504ab4d-1fc3-4116-838a-e3708400931b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h5bqj" Oct 31 03:20:57.301777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:57.301579 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6504ab4d-1fc3-4116-838a-e3708400931b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-h5bqj\" (UID: \"6504ab4d-1fc3-4116-838a-e3708400931b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h5bqj" Oct 31 03:20:57.402734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:57.402661 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc99x\" (UniqueName: \"kubernetes.io/projected/6504ab4d-1fc3-4116-838a-e3708400931b-kube-api-access-xc99x\") pod \"cert-manager-cainjector-5c57bbd4fc-h5bqj\" (UID: \"6504ab4d-1fc3-4116-838a-e3708400931b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h5bqj" Oct 31 03:20:57.403110 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:57.402780 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6504ab4d-1fc3-4116-838a-e3708400931b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-h5bqj\" (UID: \"6504ab4d-1fc3-4116-838a-e3708400931b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h5bqj" Oct 31 03:20:57.412601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:57.412565 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6504ab4d-1fc3-4116-838a-e3708400931b-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-h5bqj\" (UID: \"6504ab4d-1fc3-4116-838a-e3708400931b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h5bqj" Oct 31 03:20:57.412898 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:57.412879 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc99x\" (UniqueName: \"kubernetes.io/projected/6504ab4d-1fc3-4116-838a-e3708400931b-kube-api-access-xc99x\") pod \"cert-manager-cainjector-5c57bbd4fc-h5bqj\" (UID: \"6504ab4d-1fc3-4116-838a-e3708400931b\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h5bqj" Oct 31 03:20:57.535426 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:57.535347 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h5bqj" Oct 31 03:20:57.809027 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:57.808950 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-h5bqj"] Oct 31 03:20:57.813827 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:20:57.813798 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6504ab4d_1fc3_4116_838a_e3708400931b.slice/crio-1ddb72cd897a838a9594b7f55b69b4a49b8b33287a0ab69d4a388781f27601bf WatchSource:0}: Error finding container 1ddb72cd897a838a9594b7f55b69b4a49b8b33287a0ab69d4a388781f27601bf: Status 404 returned error can't find the container with id 1ddb72cd897a838a9594b7f55b69b4a49b8b33287a0ab69d4a388781f27601bf Oct 31 03:20:57.861632 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:57.861597 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" event={"ID":"efd854cd-40a6-4f9f-bd19-cd5d19512dba","Type":"ContainerStarted","Data":"06ab7405f52169e7bba5402b5d51a638c070b692a8fface1d507dbf85dff239e"} Oct 31 03:20:57.862833 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:57.862806 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h5bqj" event={"ID":"6504ab4d-1fc3-4116-838a-e3708400931b","Type":"ContainerStarted","Data":"1ddb72cd897a838a9594b7f55b69b4a49b8b33287a0ab69d4a388781f27601bf"} Oct 31 03:20:57.954579 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:57.954539 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:58.299725 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:58.299667 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:58.441318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:58.441280 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:58.742364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:58.742326 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:58.909209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:58.909170 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:59.077293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:59.077136 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:20:59.824571 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:20:59.824537 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:21:00.742383 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:00.742345 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:00.881768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:00.881722 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" event={"ID":"efd854cd-40a6-4f9f-bd19-cd5d19512dba","Type":"ContainerStarted","Data":"a11459fcb77198f0b80d12c2fa87b60c24bbe2c97e259d4af34010cb220aad75"} Oct 31 03:21:00.882258 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:00.881832 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" Oct 31 03:21:01.537430 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:01.537281 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:01.887952 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:01.887866 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h5bqj" event={"ID":"6504ab4d-1fc3-4116-838a-e3708400931b","Type":"ContainerStarted","Data":"56afff6d4249b9e388d09bd05e61836391431921447f990669bff8be39401735"} Oct 31 03:21:01.928369 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:01.928320 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h5bqj" podStartSLOduration=1.63476021 podStartE2EDuration="4.928306154s" totalImagesPullingTime="3.293545944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:01.927308936 +0000 UTC m=+296.462003906" watchObservedRunningTime="2026-10-31 03:21:01.928306154 +0000 UTC m=+296.463001109" Oct 31 03:21:01.929315 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:01.929282 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" podStartSLOduration=3.067191985 podStartE2EDuration="5.929270854s" totalImagesPullingTime="2.862078869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:00.898852219 +0000 UTC m=+295.433547177" watchObservedRunningTime="2026-10-31 03:21:01.929270854 +0000 UTC m=+296.463965809" Oct 31 03:21:02.322843 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:02.322805 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:02.502086 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:02.502047 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:04.473936 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:04.473886 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:04.618341 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:04.618289 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:06.780282 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:06.780244 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls"] Oct 31 03:21:06.783569 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:06.783550 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" Oct 31 03:21:06.792210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:06.792182 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c3ed8c6-5871-43dc-aef4-8f440a1060a7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bdjls\" (UID: \"3c3ed8c6-5871-43dc-aef4-8f440a1060a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" Oct 31 03:21:06.792347 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:06.792301 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp25w\" (UniqueName: \"kubernetes.io/projected/3c3ed8c6-5871-43dc-aef4-8f440a1060a7-kube-api-access-pp25w\") pod \"cert-manager-webhook-5dbbbf75cc-bdjls\" (UID: \"3c3ed8c6-5871-43dc-aef4-8f440a1060a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" Oct 31 03:21:06.793055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:06.793026 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls"] Oct 31 03:21:06.885442 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:06.885415 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6nvwh\"" Oct 31 03:21:06.892824 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:06.892785 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c3ed8c6-5871-43dc-aef4-8f440a1060a7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bdjls\" (UID: \"3c3ed8c6-5871-43dc-aef4-8f440a1060a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" Oct 31 03:21:06.892958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:06.892910 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp25w\" (UniqueName: \"kubernetes.io/projected/3c3ed8c6-5871-43dc-aef4-8f440a1060a7-kube-api-access-pp25w\") pod \"cert-manager-webhook-5dbbbf75cc-bdjls\" (UID: \"3c3ed8c6-5871-43dc-aef4-8f440a1060a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" Oct 31 03:21:06.902662 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:06.902632 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c3ed8c6-5871-43dc-aef4-8f440a1060a7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-bdjls\" (UID: \"3c3ed8c6-5871-43dc-aef4-8f440a1060a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" Oct 31 03:21:06.902779 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:06.902754 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp25w\" (UniqueName: \"kubernetes.io/projected/3c3ed8c6-5871-43dc-aef4-8f440a1060a7-kube-api-access-pp25w\") pod \"cert-manager-webhook-5dbbbf75cc-bdjls\" (UID: \"3c3ed8c6-5871-43dc-aef4-8f440a1060a7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" Oct 31 03:21:07.095869 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:07.095780 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" Oct 31 03:21:07.370923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:07.368568 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls"] Oct 31 03:21:07.374591 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:21:07.374551 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c3ed8c6_5871_43dc_aef4_8f440a1060a7.slice/crio-af5ed4fa98f1b24adf2c1630e851e383adfd78523efb8c8133dbb2e4cdaa4eaf WatchSource:0}: Error finding container af5ed4fa98f1b24adf2c1630e851e383adfd78523efb8c8133dbb2e4cdaa4eaf: Status 404 returned error can't find the container with id af5ed4fa98f1b24adf2c1630e851e383adfd78523efb8c8133dbb2e4cdaa4eaf Oct 31 03:21:07.697561 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:07.697524 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:07.844859 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:07.844818 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:07.914953 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:07.914918 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" event={"ID":"3c3ed8c6-5871-43dc-aef4-8f440a1060a7","Type":"ContainerStarted","Data":"ec00228885dd52018a1ab0bc0d3946855d0149c08a6916c4ed6781db4fbed4ba"} Oct 31 03:21:07.915101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:07.914964 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" event={"ID":"3c3ed8c6-5871-43dc-aef4-8f440a1060a7","Type":"ContainerStarted","Data":"af5ed4fa98f1b24adf2c1630e851e383adfd78523efb8c8133dbb2e4cdaa4eaf"} Oct 31 03:21:07.915101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:07.914999 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" Oct 31 03:21:07.951232 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:07.951127 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podStartSLOduration=1.951112489 podStartE2EDuration="1.951112489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:07.949962646 +0000 UTC m=+302.484657600" watchObservedRunningTime="2026-10-31 03:21:07.951112489 +0000 UTC m=+302.485807443" Oct 31 03:21:08.311408 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:08.311322 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:08.791326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:08.791293 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:09.141278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:09.141188 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:10.635487 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:10.635454 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:11.890999 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:11.890951 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" Oct 31 03:21:11.996897 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:11.996864 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:12.167307 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:12.167273 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:12.253315 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:12.253272 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-86bb8bf66f-tqrbj"] Oct 31 03:21:12.258016 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:12.257992 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-tqrbj" Oct 31 03:21:12.269579 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:12.269549 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86bb8bf66f-tqrbj"] Oct 31 03:21:12.322928 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:12.322855 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:12.329889 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:12.329860 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2bjv\" (UniqueName: \"kubernetes.io/projected/01171a38-57a8-46bb-b2a0-9d69f73f12ea-kube-api-access-z2bjv\") pod \"kube-auth-proxy-86bb8bf66f-tqrbj\" (UID: \"01171a38-57a8-46bb-b2a0-9d69f73f12ea\") " pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-tqrbj" Oct 31 03:21:12.330070 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:12.329905 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01171a38-57a8-46bb-b2a0-9d69f73f12ea-tls-certs\") pod \"kube-auth-proxy-86bb8bf66f-tqrbj\" (UID: \"01171a38-57a8-46bb-b2a0-9d69f73f12ea\") " pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-tqrbj" Oct 31 03:21:12.330070 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:12.330024 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01171a38-57a8-46bb-b2a0-9d69f73f12ea-tmp\") pod \"kube-auth-proxy-86bb8bf66f-tqrbj\" (UID: \"01171a38-57a8-46bb-b2a0-9d69f73f12ea\") " pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-tqrbj" Oct 31 03:21:12.341267 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:12.341235 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 03:21:12.341669 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:12.341621 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 03:21:12.341772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:12.341712 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pc69c\"" Oct 31 03:21:12.430986 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:12.430890 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2bjv\" (UniqueName: \"kubernetes.io/projected/01171a38-57a8-46bb-b2a0-9d69f73f12ea-kube-api-access-z2bjv\") pod \"kube-auth-proxy-86bb8bf66f-tqrbj\" (UID: \"01171a38-57a8-46bb-b2a0-9d69f73f12ea\") " pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-tqrbj" Oct 31 03:21:12.430986 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:12.430936 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01171a38-57a8-46bb-b2a0-9d69f73f12ea-tls-certs\") pod \"kube-auth-proxy-86bb8bf66f-tqrbj\" (UID: \"01171a38-57a8-46bb-b2a0-9d69f73f12ea\") " pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-tqrbj" Oct 31 03:21:12.430986 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:12.430964 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01171a38-57a8-46bb-b2a0-9d69f73f12ea-tmp\") pod \"kube-auth-proxy-86bb8bf66f-tqrbj\" (UID: \"01171a38-57a8-46bb-b2a0-9d69f73f12ea\") " pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-tqrbj" Oct 31 03:21:12.431273 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:21:12.431083 5490 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Oct 31 03:21:12.431273 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:21:12.431189 5490 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01171a38-57a8-46bb-b2a0-9d69f73f12ea-tls-certs podName:01171a38-57a8-46bb-b2a0-9d69f73f12ea nodeName:}" failed. No retries permitted until 2026-10-31 03:21:12.931164295 +0000 UTC m=+307.465859248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/01171a38-57a8-46bb-b2a0-9d69f73f12ea-tls-certs") pod "kube-auth-proxy-86bb8bf66f-tqrbj" (UID: "01171a38-57a8-46bb-b2a0-9d69f73f12ea") : secret "kube-auth-proxy-tls" not found Oct 31 03:21:12.433287 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:12.433263 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01171a38-57a8-46bb-b2a0-9d69f73f12ea-tmp\") pod \"kube-auth-proxy-86bb8bf66f-tqrbj\" (UID: \"01171a38-57a8-46bb-b2a0-9d69f73f12ea\") " pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-tqrbj" Oct 31 03:21:12.453125 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:12.453095 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2bjv\" (UniqueName: \"kubernetes.io/projected/01171a38-57a8-46bb-b2a0-9d69f73f12ea-kube-api-access-z2bjv\") pod \"kube-auth-proxy-86bb8bf66f-tqrbj\" (UID: \"01171a38-57a8-46bb-b2a0-9d69f73f12ea\") " pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-tqrbj" Oct 31 03:21:12.635247 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:12.635213 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:12.873050 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:12.872958 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:12.935860 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:12.935828 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01171a38-57a8-46bb-b2a0-9d69f73f12ea-tls-certs\") pod \"kube-auth-proxy-86bb8bf66f-tqrbj\" (UID: \"01171a38-57a8-46bb-b2a0-9d69f73f12ea\") " pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-tqrbj" Oct 31 03:21:12.938525 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:12.938493 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01171a38-57a8-46bb-b2a0-9d69f73f12ea-tls-certs\") pod \"kube-auth-proxy-86bb8bf66f-tqrbj\" (UID: \"01171a38-57a8-46bb-b2a0-9d69f73f12ea\") " pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-tqrbj" Oct 31 03:21:13.069408 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:13.069366 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:13.171452 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:13.171420 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-tqrbj" Oct 31 03:21:13.454278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:13.454251 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86bb8bf66f-tqrbj"] Oct 31 03:21:13.457167 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:21:13.457135 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01171a38_57a8_46bb_b2a0_9d69f73f12ea.slice/crio-9b0babccbfadb07f6034212063de959e287b26cd8112c119a93d10f085325b82 WatchSource:0}: Error finding container 9b0babccbfadb07f6034212063de959e287b26cd8112c119a93d10f085325b82: Status 404 returned error can't find the container with id 9b0babccbfadb07f6034212063de959e287b26cd8112c119a93d10f085325b82 Oct 31 03:21:13.664947 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:13.664910 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:13.921435 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:13.921404 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" Oct 31 03:21:13.944986 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:13.944955 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-tqrbj" event={"ID":"01171a38-57a8-46bb-b2a0-9d69f73f12ea","Type":"ContainerStarted","Data":"9b0babccbfadb07f6034212063de959e287b26cd8112c119a93d10f085325b82"} Oct 31 03:21:14.258837 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.258746 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:14.271055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.271022 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z"] Oct 31 03:21:14.274606 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.274581 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" Oct 31 03:21:14.284919 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.284894 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-skcgk\"" Oct 31 03:21:14.285040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.284973 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 03:21:14.285136 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.285119 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 03:21:14.287793 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.287760 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 31 03:21:14.292295 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.292272 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z"] Oct 31 03:21:14.447265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.447227 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/30874c80-e957-4c9a-ab8f-0616de567c3d-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-wlb6z\" (UID: \"30874c80-e957-4c9a-ab8f-0616de567c3d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" Oct 31 03:21:14.447495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.447292 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30874c80-e957-4c9a-ab8f-0616de567c3d-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-wlb6z\" (UID: \"30874c80-e957-4c9a-ab8f-0616de567c3d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" Oct 31 03:21:14.447495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.447347 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8c8l\" (UniqueName: \"kubernetes.io/projected/30874c80-e957-4c9a-ab8f-0616de567c3d-kube-api-access-p8c8l\") pod \"kserve-module-controller-manager-7566cf5fc8-wlb6z\" (UID: \"30874c80-e957-4c9a-ab8f-0616de567c3d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" Oct 31 03:21:14.447495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.447392 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30874c80-e957-4c9a-ab8f-0616de567c3d-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-wlb6z\" (UID: \"30874c80-e957-4c9a-ab8f-0616de567c3d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" Oct 31 03:21:14.548786 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.548701 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30874c80-e957-4c9a-ab8f-0616de567c3d-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-wlb6z\" (UID: \"30874c80-e957-4c9a-ab8f-0616de567c3d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" Oct 31 03:21:14.548786 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.548766 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/30874c80-e957-4c9a-ab8f-0616de567c3d-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-wlb6z\" (UID: \"30874c80-e957-4c9a-ab8f-0616de567c3d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" Oct 31 03:21:14.548987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.548796 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30874c80-e957-4c9a-ab8f-0616de567c3d-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-wlb6z\" (UID: \"30874c80-e957-4c9a-ab8f-0616de567c3d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" Oct 31 03:21:14.548987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.548853 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8c8l\" (UniqueName: \"kubernetes.io/projected/30874c80-e957-4c9a-ab8f-0616de567c3d-kube-api-access-p8c8l\") pod \"kserve-module-controller-manager-7566cf5fc8-wlb6z\" (UID: \"30874c80-e957-4c9a-ab8f-0616de567c3d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" Oct 31 03:21:14.549141 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.549121 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30874c80-e957-4c9a-ab8f-0616de567c3d-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-wlb6z\" (UID: \"30874c80-e957-4c9a-ab8f-0616de567c3d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" Oct 31 03:21:14.549427 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.549407 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/30874c80-e957-4c9a-ab8f-0616de567c3d-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-wlb6z\" (UID: \"30874c80-e957-4c9a-ab8f-0616de567c3d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" Oct 31 03:21:14.550225 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.550198 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/30874c80-e957-4c9a-ab8f-0616de567c3d-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-wlb6z\" (UID: \"30874c80-e957-4c9a-ab8f-0616de567c3d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" Oct 31 03:21:14.558319 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.558290 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8c8l\" (UniqueName: \"kubernetes.io/projected/30874c80-e957-4c9a-ab8f-0616de567c3d-kube-api-access-p8c8l\") pod \"kserve-module-controller-manager-7566cf5fc8-wlb6z\" (UID: \"30874c80-e957-4c9a-ab8f-0616de567c3d\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" Oct 31 03:21:14.593972 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.593929 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" Oct 31 03:21:14.729991 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.729958 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:14.901129 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.901020 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z"] Oct 31 03:21:14.903743 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:21:14.903709 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30874c80_e957_4c9a_ab8f_0616de567c3d.slice/crio-2bf285dd30a208c361841330a882cd86d30b9dff4ac81e9223ce7cd61784e943 WatchSource:0}: Error finding container 2bf285dd30a208c361841330a882cd86d30b9dff4ac81e9223ce7cd61784e943: Status 404 returned error can't find the container with id 2bf285dd30a208c361841330a882cd86d30b9dff4ac81e9223ce7cd61784e943 Oct 31 03:21:14.904869 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.904839 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:14.952589 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:14.952554 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" event={"ID":"30874c80-e957-4c9a-ab8f-0616de567c3d","Type":"ContainerStarted","Data":"2bf285dd30a208c361841330a882cd86d30b9dff4ac81e9223ce7cd61784e943"} Oct 31 03:21:15.194562 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:15.194523 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:15.376123 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:15.376085 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:16.964340 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:16.964297 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-tqrbj" event={"ID":"01171a38-57a8-46bb-b2a0-9d69f73f12ea","Type":"ContainerStarted","Data":"43cacbdadf15b1d02edfaa5bf387d5344cb94c2cda4e5d4c2d476ec3e6923194"} Oct 31 03:21:16.982377 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:16.982325 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-86bb8bf66f-tqrbj" podStartSLOduration=2.129696314 podStartE2EDuration="4.982311172s" totalImagesPullingTime="2.852614858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:16.98108622 +0000 UTC m=+311.515781178" watchObservedRunningTime="2026-10-31 03:21:16.982311172 +0000 UTC m=+311.517006141" Oct 31 03:21:17.127198 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:21:17.127154 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:21:17.369760 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:17.369723 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:17.512538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:17.512454 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:17.723035 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:17.722995 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:17.876357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:17.876269 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:17.970670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:17.970607 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" event={"ID":"30874c80-e957-4c9a-ab8f-0616de567c3d","Type":"ContainerStarted","Data":"6e71c74b1cffbbe30ddc2c582fd23df1f5369ed32ca47d10dc57c99bf8312729"} Oct 31 03:21:17.971143 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:17.970839 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" Oct 31 03:21:17.990894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:17.990841 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" podStartSLOduration=1.741419842 podStartE2EDuration="3.990826158s" totalImagesPullingTime="2.249406316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:17.988816551 +0000 UTC m=+312.523511505" watchObservedRunningTime="2026-10-31 03:21:17.990826158 +0000 UTC m=+312.525521113" Oct 31 03:21:23.716416 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:23.716381 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:23.850770 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:23.850734 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:23.976228 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:23.976143 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:24.207224 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:24.207189 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:24.342397 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:24.342308 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:31.722453 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:31.722414 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:31.868849 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:31.868802 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:34.279051 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:34.279009 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:34.424708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:34.424664 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:36.719926 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:36.719890 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:36.844055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:36.844014 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:38.974844 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:38.974810 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:38.976022 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:38.975999 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" Oct 31 03:21:39.099655 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:39.099611 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:42.124822 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:42.124785 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:42.316110 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:42.316076 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:42.485242 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:42.485205 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:47.126485 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:21:47.126441 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:21:50.762839 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:50.762799 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:50.971426 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:50.971389 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:53.363556 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:53.363514 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:53.500128 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:53.500087 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:56.364967 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:56.364923 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:56.495566 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:56.495530 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:59.471546 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:59.471513 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-qh74f"] Oct 31 03:21:59.475804 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:59.475784 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-qh74f" Oct 31 03:21:59.482427 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:59.482404 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-qh74f"] Oct 31 03:21:59.483978 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:59.483961 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fzbgx\"" Oct 31 03:21:59.492837 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:59.491751 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 03:21:59.529608 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:59.529547 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:59.566467 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:59.566436 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-qh74f"] Oct 31 03:21:59.637806 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:59.637765 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7fc7393b-71f1-4f1e-af41-56eb647348f8-config-file\") pod \"limitador-limitador-74dfcfb894-qh74f\" (UID: \"7fc7393b-71f1-4f1e-af41-56eb647348f8\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-qh74f" Oct 31 03:21:59.637995 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:59.637840 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcctl\" (UniqueName: \"kubernetes.io/projected/7fc7393b-71f1-4f1e-af41-56eb647348f8-kube-api-access-bcctl\") pod \"limitador-limitador-74dfcfb894-qh74f\" (UID: \"7fc7393b-71f1-4f1e-af41-56eb647348f8\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-qh74f" Oct 31 03:21:59.671507 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:59.671469 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:21:59.739325 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:59.739223 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7fc7393b-71f1-4f1e-af41-56eb647348f8-config-file\") pod \"limitador-limitador-74dfcfb894-qh74f\" (UID: \"7fc7393b-71f1-4f1e-af41-56eb647348f8\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-qh74f" Oct 31 03:21:59.739325 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:59.739311 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcctl\" (UniqueName: \"kubernetes.io/projected/7fc7393b-71f1-4f1e-af41-56eb647348f8-kube-api-access-bcctl\") pod \"limitador-limitador-74dfcfb894-qh74f\" (UID: \"7fc7393b-71f1-4f1e-af41-56eb647348f8\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-qh74f" Oct 31 03:21:59.740012 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:59.739988 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7fc7393b-71f1-4f1e-af41-56eb647348f8-config-file\") pod \"limitador-limitador-74dfcfb894-qh74f\" (UID: \"7fc7393b-71f1-4f1e-af41-56eb647348f8\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-qh74f" Oct 31 03:21:59.751931 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:59.751905 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcctl\" (UniqueName: \"kubernetes.io/projected/7fc7393b-71f1-4f1e-af41-56eb647348f8-kube-api-access-bcctl\") pod \"limitador-limitador-74dfcfb894-qh74f\" (UID: \"7fc7393b-71f1-4f1e-af41-56eb647348f8\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-qh74f" Oct 31 03:21:59.789805 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:59.789772 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-qh74f" Oct 31 03:21:59.802230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:21:59.802197 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:00.066731 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:00.066555 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-qh74f"] Oct 31 03:22:00.068929 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:22:00.068906 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc7393b_71f1_4f1e_af41_56eb647348f8.slice/crio-4d8d4016f2d93ab6180db475bc173560ac4bf5761749a4caa416959081350373 WatchSource:0}: Error finding container 4d8d4016f2d93ab6180db475bc173560ac4bf5761749a4caa416959081350373: Status 404 returned error can't find the container with id 4d8d4016f2d93ab6180db475bc173560ac4bf5761749a4caa416959081350373 Oct 31 03:22:00.165714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:00.165667 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-qh74f" event={"ID":"7fc7393b-71f1-4f1e-af41-56eb647348f8","Type":"ContainerStarted","Data":"4d8d4016f2d93ab6180db475bc173560ac4bf5761749a4caa416959081350373"} Oct 31 03:22:00.274126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:00.274092 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-j8ww9"] Oct 31 03:22:00.278180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:00.278162 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-j8ww9" Oct 31 03:22:00.283184 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:00.283157 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b2rkt\"" Oct 31 03:22:00.284871 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:00.284847 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-j8ww9"] Oct 31 03:22:00.385056 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:00.384976 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:00.446470 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:00.446435 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89chm\" (UniqueName: \"kubernetes.io/projected/0d9813a7-6395-403c-9030-595d7bb5768e-kube-api-access-89chm\") pod \"authorino-57f7f8659f-j8ww9\" (UID: \"0d9813a7-6395-403c-9030-595d7bb5768e\") " pod="kuadrant-system/authorino-57f7f8659f-j8ww9" Oct 31 03:22:00.532176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:00.532138 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:00.547167 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:00.547131 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89chm\" (UniqueName: \"kubernetes.io/projected/0d9813a7-6395-403c-9030-595d7bb5768e-kube-api-access-89chm\") pod \"authorino-57f7f8659f-j8ww9\" (UID: \"0d9813a7-6395-403c-9030-595d7bb5768e\") " pod="kuadrant-system/authorino-57f7f8659f-j8ww9" Oct 31 03:22:00.556538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:00.556508 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89chm\" (UniqueName: \"kubernetes.io/projected/0d9813a7-6395-403c-9030-595d7bb5768e-kube-api-access-89chm\") pod \"authorino-57f7f8659f-j8ww9\" (UID: \"0d9813a7-6395-403c-9030-595d7bb5768e\") " pod="kuadrant-system/authorino-57f7f8659f-j8ww9" Oct 31 03:22:00.591615 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:00.591585 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-j8ww9" Oct 31 03:22:00.861376 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:00.861317 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-j8ww9"] Oct 31 03:22:00.863853 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:22:00.863825 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d9813a7_6395_403c_9030_595d7bb5768e.slice/crio-06dfe92a5329276dc8a767406f7d910e04d7f3b45e3a709606eb371aeb14177a WatchSource:0}: Error finding container 06dfe92a5329276dc8a767406f7d910e04d7f3b45e3a709606eb371aeb14177a: Status 404 returned error can't find the container with id 06dfe92a5329276dc8a767406f7d910e04d7f3b45e3a709606eb371aeb14177a Oct 31 03:22:01.160562 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:01.160506 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:01.171286 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:01.171248 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-j8ww9" event={"ID":"0d9813a7-6395-403c-9030-595d7bb5768e","Type":"ContainerStarted","Data":"06dfe92a5329276dc8a767406f7d910e04d7f3b45e3a709606eb371aeb14177a"} Oct 31 03:22:01.299903 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:01.299862 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:02.179202 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:02.179162 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-qh74f" event={"ID":"7fc7393b-71f1-4f1e-af41-56eb647348f8","Type":"ContainerStarted","Data":"739c35df35f857d52435661b6cf886b52c925a937e4da17c20b4251ff1108398"} Oct 31 03:22:02.179776 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:02.179296 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-qh74f" Oct 31 03:22:02.245635 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:02.245596 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:02.392239 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:02.392206 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:03.035127 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:03.035089 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:03.186057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:03.185965 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-j8ww9" event={"ID":"0d9813a7-6395-403c-9030-595d7bb5768e","Type":"ContainerStarted","Data":"fcc79583c12a00712ba2d836daad55c0ba5c7b031da225043754ee685534dc6a"} Oct 31 03:22:03.214708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:03.214627 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-qh74f" podStartSLOduration=2.684339047 podStartE2EDuration="4.214610337s" totalImagesPullingTime="1.53027129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:02.204866029 +0000 UTC m=+356.739560986" watchObservedRunningTime="2026-10-31 03:22:03.214610337 +0000 UTC m=+357.749305292" Oct 31 03:22:03.216133 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:03.216090 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-j8ww9" podStartSLOduration=1.148545931 podStartE2EDuration="3.216080669s" totalImagesPullingTime="2.067534738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:03.21410308 +0000 UTC m=+357.748798034" watchObservedRunningTime="2026-10-31 03:22:03.216080669 +0000 UTC m=+357.750775829" Oct 31 03:22:03.216666 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:03.216636 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:05.408775 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:05.408740 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-j8ww9"] Oct 31 03:22:05.409211 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:05.408899 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-j8ww9" podUID="0d9813a7-6395-403c-9030-595d7bb5768e" containerName="authorino" containerID="cri-o://fcc79583c12a00712ba2d836daad55c0ba5c7b031da225043754ee685534dc6a" gracePeriod=30 Oct 31 03:22:05.666337 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:05.666255 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-j8ww9" Oct 31 03:22:05.688347 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:05.688317 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d9813a7-6395-403c-9030-595d7bb5768e-kube-api-access-89chm\" (UniqueName: \"kubernetes.io/projected/0d9813a7-6395-403c-9030-595d7bb5768e-kube-api-access-89chm\") pod \"0d9813a7-6395-403c-9030-595d7bb5768e\" (UID: \"0d9813a7-6395-403c-9030-595d7bb5768e\") " Oct 31 03:22:05.693080 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:05.693050 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d9813a7-6395-403c-9030-595d7bb5768e-kube-api-access-89chm" pod "0d9813a7-6395-403c-9030-595d7bb5768e" (UID: "0d9813a7-6395-403c-9030-595d7bb5768e"). InnerVolumeSpecName "kube-api-access-89chm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:05.788931 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:05.788900 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89chm\" (UniqueName: \"kubernetes.io/projected/0d9813a7-6395-403c-9030-595d7bb5768e-kube-api-access-89chm\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:06.200726 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:06.200672 5490 generic.go:356] "Generic (PLEG): container finished" podID="0d9813a7-6395-403c-9030-595d7bb5768e" containerID="fcc79583c12a00712ba2d836daad55c0ba5c7b031da225043754ee685534dc6a" exitCode=0 Oct 31 03:22:06.200726 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:06.200712 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-j8ww9" event={"ID":"0d9813a7-6395-403c-9030-595d7bb5768e","Type":"ContainerDied","Data":"fcc79583c12a00712ba2d836daad55c0ba5c7b031da225043754ee685534dc6a"} Oct 31 03:22:06.200726 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:06.200730 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-j8ww9" Oct 31 03:22:06.201041 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:06.200763 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-j8ww9" event={"ID":"0d9813a7-6395-403c-9030-595d7bb5768e","Type":"ContainerDied","Data":"06dfe92a5329276dc8a767406f7d910e04d7f3b45e3a709606eb371aeb14177a"} Oct 31 03:22:06.201041 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:06.200785 5490 scope.go:122] "RemoveContainer" containerID="fcc79583c12a00712ba2d836daad55c0ba5c7b031da225043754ee685534dc6a" Oct 31 03:22:06.210121 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:06.210102 5490 scope.go:122] "RemoveContainer" containerID="fcc79583c12a00712ba2d836daad55c0ba5c7b031da225043754ee685534dc6a" Oct 31 03:22:06.210406 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:22:06.210388 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc79583c12a00712ba2d836daad55c0ba5c7b031da225043754ee685534dc6a\": container with ID starting with fcc79583c12a00712ba2d836daad55c0ba5c7b031da225043754ee685534dc6a not found: ID does not exist" containerID="fcc79583c12a00712ba2d836daad55c0ba5c7b031da225043754ee685534dc6a" Oct 31 03:22:06.210447 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:06.210417 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc79583c12a00712ba2d836daad55c0ba5c7b031da225043754ee685534dc6a"} err="failed to get container status \"fcc79583c12a00712ba2d836daad55c0ba5c7b031da225043754ee685534dc6a\": rpc error: code = NotFound desc = could not find container \"fcc79583c12a00712ba2d836daad55c0ba5c7b031da225043754ee685534dc6a\": container with ID starting with fcc79583c12a00712ba2d836daad55c0ba5c7b031da225043754ee685534dc6a not found: ID does not exist" Oct 31 03:22:06.226245 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:06.226216 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-j8ww9"] Oct 31 03:22:06.233978 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:06.233951 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-j8ww9"] Oct 31 03:22:07.573022 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:07.572987 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d9813a7-6395-403c-9030-595d7bb5768e" path="/var/lib/kubelet/pods/0d9813a7-6395-403c-9030-595d7bb5768e/volumes" Oct 31 03:22:08.591116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.591081 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:08.654865 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.654831 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-wnnv7"] Oct 31 03:22:08.655245 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.655227 5490 cpu_manager.go:414] "containerMap: removing container" podUID="0d9813a7-6395-403c-9030-595d7bb5768e" containerName="authorino" Oct 31 03:22:08.655324 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.655247 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d9813a7-6395-403c-9030-595d7bb5768e" containerName="authorino" Oct 31 03:22:08.655372 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.655327 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d9813a7-6395-403c-9030-595d7bb5768e" containerName="authorino" Oct 31 03:22:08.660799 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.660772 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" Oct 31 03:22:08.670103 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.670072 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-wnnv7"] Oct 31 03:22:08.672666 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.672645 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 31 03:22:08.672889 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.672778 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-lz9fj\"" Oct 31 03:22:08.709886 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.709850 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce529bc9-a44d-4808-8d2c-996e14f88a54-cert\") pod \"kserve-controller-manager-776cc66d47-wnnv7\" (UID: \"ce529bc9-a44d-4808-8d2c-996e14f88a54\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" Oct 31 03:22:08.709886 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.709890 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tb7q\" (UniqueName: \"kubernetes.io/projected/ce529bc9-a44d-4808-8d2c-996e14f88a54-kube-api-access-2tb7q\") pod \"kserve-controller-manager-776cc66d47-wnnv7\" (UID: \"ce529bc9-a44d-4808-8d2c-996e14f88a54\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" Oct 31 03:22:08.721720 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.721688 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-6r4mx"] Oct 31 03:22:08.725166 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.725144 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-6r4mx" Oct 31 03:22:08.739106 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.739082 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-6r4mx"] Oct 31 03:22:08.743430 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.743403 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:08.777962 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.777936 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-xgnxw\"" Oct 31 03:22:08.778111 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.777966 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 31 03:22:08.778111 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.778049 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 31 03:22:08.811645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.810839 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-cert\") pod \"odh-model-controller-769875db56-6r4mx\" (UID: \"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2\") " pod="opendatahub/odh-model-controller-769875db56-6r4mx" Oct 31 03:22:08.811645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.810939 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce529bc9-a44d-4808-8d2c-996e14f88a54-cert\") pod \"kserve-controller-manager-776cc66d47-wnnv7\" (UID: \"ce529bc9-a44d-4808-8d2c-996e14f88a54\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" Oct 31 03:22:08.811645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.810970 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-metrics-certs\") pod \"odh-model-controller-769875db56-6r4mx\" (UID: \"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2\") " pod="opendatahub/odh-model-controller-769875db56-6r4mx" Oct 31 03:22:08.811645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.811021 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tb7q\" (UniqueName: \"kubernetes.io/projected/ce529bc9-a44d-4808-8d2c-996e14f88a54-kube-api-access-2tb7q\") pod \"kserve-controller-manager-776cc66d47-wnnv7\" (UID: \"ce529bc9-a44d-4808-8d2c-996e14f88a54\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" Oct 31 03:22:08.811645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.811078 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n87jc\" (UniqueName: \"kubernetes.io/projected/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-kube-api-access-n87jc\") pod \"odh-model-controller-769875db56-6r4mx\" (UID: \"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2\") " pod="opendatahub/odh-model-controller-769875db56-6r4mx" Oct 31 03:22:08.818296 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.818266 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce529bc9-a44d-4808-8d2c-996e14f88a54-cert\") pod \"kserve-controller-manager-776cc66d47-wnnv7\" (UID: \"ce529bc9-a44d-4808-8d2c-996e14f88a54\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" Oct 31 03:22:08.820951 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.820926 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tb7q\" (UniqueName: \"kubernetes.io/projected/ce529bc9-a44d-4808-8d2c-996e14f88a54-kube-api-access-2tb7q\") pod \"kserve-controller-manager-776cc66d47-wnnv7\" (UID: \"ce529bc9-a44d-4808-8d2c-996e14f88a54\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" Oct 31 03:22:08.912004 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.911965 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-metrics-certs\") pod \"odh-model-controller-769875db56-6r4mx\" (UID: \"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2\") " pod="opendatahub/odh-model-controller-769875db56-6r4mx" Oct 31 03:22:08.912166 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.912025 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n87jc\" (UniqueName: \"kubernetes.io/projected/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-kube-api-access-n87jc\") pod \"odh-model-controller-769875db56-6r4mx\" (UID: \"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2\") " pod="opendatahub/odh-model-controller-769875db56-6r4mx" Oct 31 03:22:08.912166 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.912078 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-cert\") pod \"odh-model-controller-769875db56-6r4mx\" (UID: \"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2\") " pod="opendatahub/odh-model-controller-769875db56-6r4mx" Oct 31 03:22:08.914430 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.914397 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-cert\") pod \"odh-model-controller-769875db56-6r4mx\" (UID: \"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2\") " pod="opendatahub/odh-model-controller-769875db56-6r4mx" Oct 31 03:22:08.914539 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.914521 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-metrics-certs\") pod \"odh-model-controller-769875db56-6r4mx\" (UID: \"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2\") " pod="opendatahub/odh-model-controller-769875db56-6r4mx" Oct 31 03:22:08.922462 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.922436 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n87jc\" (UniqueName: \"kubernetes.io/projected/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-kube-api-access-n87jc\") pod \"odh-model-controller-769875db56-6r4mx\" (UID: \"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2\") " pod="opendatahub/odh-model-controller-769875db56-6r4mx" Oct 31 03:22:08.975407 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.975368 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" Oct 31 03:22:08.992373 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:08.992340 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:09.039959 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:09.039886 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-6r4mx" Oct 31 03:22:09.195360 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:09.195270 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:09.251818 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:09.251795 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-wnnv7"] Oct 31 03:22:09.253729 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:22:09.253703 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce529bc9_a44d_4808_8d2c_996e14f88a54.slice/crio-ebea2c65a3c1ba9bbede4af8a606c1003f86630642098a0a1073139abafe52b0 WatchSource:0}: Error finding container ebea2c65a3c1ba9bbede4af8a606c1003f86630642098a0a1073139abafe52b0: Status 404 returned error can't find the container with id ebea2c65a3c1ba9bbede4af8a606c1003f86630642098a0a1073139abafe52b0 Oct 31 03:22:09.338155 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:09.338127 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-6r4mx"] Oct 31 03:22:09.339270 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:22:09.339236 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f0e9f6_2e01_4338_8243_346ffb2c7cc2.slice/crio-5849752547eed196e235e4809d34f2d225728b395f98ec7bba9d29a9814af3c0 WatchSource:0}: Error finding container 5849752547eed196e235e4809d34f2d225728b395f98ec7bba9d29a9814af3c0: Status 404 returned error can't find the container with id 5849752547eed196e235e4809d34f2d225728b395f98ec7bba9d29a9814af3c0 Oct 31 03:22:09.952460 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:09.952420 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:10.119073 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:10.119025 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:10.227666 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:10.227553 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-6r4mx" event={"ID":"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2","Type":"ContainerStarted","Data":"5849752547eed196e235e4809d34f2d225728b395f98ec7bba9d29a9814af3c0"} Oct 31 03:22:10.229939 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:10.229885 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" event={"ID":"ce529bc9-a44d-4808-8d2c-996e14f88a54","Type":"ContainerStarted","Data":"ebea2c65a3c1ba9bbede4af8a606c1003f86630642098a0a1073139abafe52b0"} Oct 31 03:22:10.359129 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:10.359092 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:12.249453 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:12.249412 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" event={"ID":"ce529bc9-a44d-4808-8d2c-996e14f88a54","Type":"ContainerStarted","Data":"5acd6599b0c4d6104a0d49fefaf92afa22f688ec7ddf1032827396daae057dce"} Oct 31 03:22:12.249921 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:12.249691 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" Oct 31 03:22:13.187480 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:13.187446 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-qh74f" Oct 31 03:22:13.207046 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:13.206981 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podStartSLOduration=2.624031911 podStartE2EDuration="5.206962265s" totalImagesPullingTime="2.582930354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:12.267792027 +0000 UTC m=+366.802486983" watchObservedRunningTime="2026-10-31 03:22:13.206962265 +0000 UTC m=+367.741657220" Oct 31 03:22:13.259607 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:13.259566 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-6r4mx" event={"ID":"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2","Type":"ContainerStarted","Data":"7209ae60ec915af7c0e7e97853a9c90448c6f22efc9001bbe9ec2bededab41a1"} Oct 31 03:22:13.260067 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:13.259635 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-6r4mx" Oct 31 03:22:13.277909 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:13.277715 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-6r4mx" podStartSLOduration=2.247627986 podStartE2EDuration="5.277695952s" totalImagesPullingTime="3.030067966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:13.276994364 +0000 UTC m=+367.811689321" watchObservedRunningTime="2026-10-31 03:22:13.277695952 +0000 UTC m=+367.812390901" Oct 31 03:22:13.758777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:13.758739 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-qh74f"] Oct 31 03:22:13.758984 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:13.758930 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-qh74f" podUID="7fc7393b-71f1-4f1e-af41-56eb647348f8" containerName="limitador" containerID="cri-o://739c35df35f857d52435661b6cf886b52c925a937e4da17c20b4251ff1108398" gracePeriod=30 Oct 31 03:22:14.266090 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:14.266044 5490 generic.go:356] "Generic (PLEG): container finished" podID="7fc7393b-71f1-4f1e-af41-56eb647348f8" containerID="739c35df35f857d52435661b6cf886b52c925a937e4da17c20b4251ff1108398" exitCode=0 Oct 31 03:22:14.266497 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:14.266120 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-qh74f" event={"ID":"7fc7393b-71f1-4f1e-af41-56eb647348f8","Type":"ContainerDied","Data":"739c35df35f857d52435661b6cf886b52c925a937e4da17c20b4251ff1108398"} Oct 31 03:22:14.310901 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:14.310871 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-qh74f" Oct 31 03:22:14.363972 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:14.363883 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc7393b-71f1-4f1e-af41-56eb647348f8-kube-api-access-bcctl\" (UniqueName: \"kubernetes.io/projected/7fc7393b-71f1-4f1e-af41-56eb647348f8-kube-api-access-bcctl\") pod \"7fc7393b-71f1-4f1e-af41-56eb647348f8\" (UID: \"7fc7393b-71f1-4f1e-af41-56eb647348f8\") " Oct 31 03:22:14.364122 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:14.364012 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc7393b-71f1-4f1e-af41-56eb647348f8-config-file\" (UniqueName: \"kubernetes.io/configmap/7fc7393b-71f1-4f1e-af41-56eb647348f8-config-file\") pod \"7fc7393b-71f1-4f1e-af41-56eb647348f8\" (UID: \"7fc7393b-71f1-4f1e-af41-56eb647348f8\") " Oct 31 03:22:14.364446 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:14.364405 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc7393b-71f1-4f1e-af41-56eb647348f8-config-file" pod "7fc7393b-71f1-4f1e-af41-56eb647348f8" (UID: "7fc7393b-71f1-4f1e-af41-56eb647348f8"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:22:14.366459 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:14.366438 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc7393b-71f1-4f1e-af41-56eb647348f8-kube-api-access-bcctl" pod "7fc7393b-71f1-4f1e-af41-56eb647348f8" (UID: "7fc7393b-71f1-4f1e-af41-56eb647348f8"). InnerVolumeSpecName "kube-api-access-bcctl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:14.465036 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:14.464995 5490 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7fc7393b-71f1-4f1e-af41-56eb647348f8-config-file\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:14.465036 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:14.465032 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcctl\" (UniqueName: \"kubernetes.io/projected/7fc7393b-71f1-4f1e-af41-56eb647348f8-kube-api-access-bcctl\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:15.234508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:15.234471 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:15.271882 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:15.271850 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-qh74f" event={"ID":"7fc7393b-71f1-4f1e-af41-56eb647348f8","Type":"ContainerDied","Data":"4d8d4016f2d93ab6180db475bc173560ac4bf5761749a4caa416959081350373"} Oct 31 03:22:15.272288 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:15.271888 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-qh74f" Oct 31 03:22:15.272288 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:15.271910 5490 scope.go:122] "RemoveContainer" containerID="739c35df35f857d52435661b6cf886b52c925a937e4da17c20b4251ff1108398" Oct 31 03:22:15.292311 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:15.292282 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-qh74f"] Oct 31 03:22:15.296808 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:15.296785 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-qh74f"] Oct 31 03:22:15.577236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:15.577159 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc7393b-71f1-4f1e-af41-56eb647348f8" path="/var/lib/kubelet/pods/7fc7393b-71f1-4f1e-af41-56eb647348f8/volumes" Oct 31 03:22:15.706995 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:15.706954 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:15.853155 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:15.853054 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:16.360049 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:16.360010 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb"] Oct 31 03:22:16.360428 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:16.360367 5490 cpu_manager.go:414] "containerMap: removing container" podUID="7fc7393b-71f1-4f1e-af41-56eb647348f8" containerName="limitador" Oct 31 03:22:16.360428 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:16.360379 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc7393b-71f1-4f1e-af41-56eb647348f8" containerName="limitador" Oct 31 03:22:16.360498 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:16.360448 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc7393b-71f1-4f1e-af41-56eb647348f8" containerName="limitador" Oct 31 03:22:16.366365 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:16.366340 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" Oct 31 03:22:16.376607 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:16.376585 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 03:22:16.377344 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:16.377316 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 03:22:16.378182 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:16.378153 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb"] Oct 31 03:22:16.455477 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:16.455433 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 31 03:22:16.455652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:16.455534 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-gznwj\"" Oct 31 03:22:16.482089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:16.482052 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/85995108-6b7b-4e56-8260-00b5347d6e6a-data\") pod \"postgres-7cdbf6dc44-82ctb\" (UID: \"85995108-6b7b-4e56-8260-00b5347d6e6a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" Oct 31 03:22:16.482237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:16.482114 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jf7b\" (UniqueName: \"kubernetes.io/projected/85995108-6b7b-4e56-8260-00b5347d6e6a-kube-api-access-9jf7b\") pod \"postgres-7cdbf6dc44-82ctb\" (UID: \"85995108-6b7b-4e56-8260-00b5347d6e6a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" Oct 31 03:22:16.486089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:16.486062 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:16.583108 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:16.583076 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jf7b\" (UniqueName: \"kubernetes.io/projected/85995108-6b7b-4e56-8260-00b5347d6e6a-kube-api-access-9jf7b\") pod \"postgres-7cdbf6dc44-82ctb\" (UID: \"85995108-6b7b-4e56-8260-00b5347d6e6a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" Oct 31 03:22:16.583277 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:16.583142 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/85995108-6b7b-4e56-8260-00b5347d6e6a-data\") pod \"postgres-7cdbf6dc44-82ctb\" (UID: \"85995108-6b7b-4e56-8260-00b5347d6e6a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" Oct 31 03:22:16.583486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:16.583469 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/85995108-6b7b-4e56-8260-00b5347d6e6a-data\") pod \"postgres-7cdbf6dc44-82ctb\" (UID: \"85995108-6b7b-4e56-8260-00b5347d6e6a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" Oct 31 03:22:16.592234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:16.592206 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jf7b\" (UniqueName: \"kubernetes.io/projected/85995108-6b7b-4e56-8260-00b5347d6e6a-kube-api-access-9jf7b\") pod \"postgres-7cdbf6dc44-82ctb\" (UID: \"85995108-6b7b-4e56-8260-00b5347d6e6a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" Oct 31 03:22:16.680624 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:16.680587 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" Oct 31 03:22:16.956442 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:16.956364 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb"] Oct 31 03:22:16.959873 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:22:16.959838 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85995108_6b7b_4e56_8260_00b5347d6e6a.slice/crio-5094e38a59f15a4983336cec98b50505a5918917b934d52ac99fc9b101dbe3f2 WatchSource:0}: Error finding container 5094e38a59f15a4983336cec98b50505a5918917b934d52ac99fc9b101dbe3f2: Status 404 returned error can't find the container with id 5094e38a59f15a4983336cec98b50505a5918917b934d52ac99fc9b101dbe3f2 Oct 31 03:22:17.126336 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:22:17.126295 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:22:17.283380 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:17.283295 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" event={"ID":"85995108-6b7b-4e56-8260-00b5347d6e6a","Type":"ContainerStarted","Data":"5094e38a59f15a4983336cec98b50505a5918917b934d52ac99fc9b101dbe3f2"} Oct 31 03:22:24.268427 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:24.268392 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-6r4mx" Oct 31 03:22:24.320733 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:24.320695 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" event={"ID":"85995108-6b7b-4e56-8260-00b5347d6e6a","Type":"ContainerStarted","Data":"90bf6c8766538c20745376c108b1682ab85ea75e1b90ddb2060c94b9807259ce"} Oct 31 03:22:24.320906 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:24.320762 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" Oct 31 03:22:24.338604 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:24.338545 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" podStartSLOduration=1.9655207620000001 podStartE2EDuration="8.338527931s" totalImagesPullingTime="6.373007169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:24.337595495 +0000 UTC m=+378.872290451" watchObservedRunningTime="2026-10-31 03:22:24.338527931 +0000 UTC m=+378.873222885" Oct 31 03:22:30.355069 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:30.355035 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" Oct 31 03:22:37.504773 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.504737 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-f9vsn"] Oct 31 03:22:37.508332 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.508315 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-f9vsn" Oct 31 03:22:37.523446 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.523416 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 03:22:37.523795 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.523773 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 03:22:37.524910 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.524884 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-f9vsn"] Oct 31 03:22:37.526566 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.526548 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cjzft\"" Oct 31 03:22:37.546088 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.546052 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:37.572040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.571998 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 03:22:37.578592 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.578571 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8wfs\" (UniqueName: \"kubernetes.io/projected/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-kube-api-access-p8wfs\") pod \"maas-controller-579cf7c6d-f9vsn\" (UID: \"278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902\") " pod="opendatahub/maas-controller-579cf7c6d-f9vsn" Oct 31 03:22:37.578722 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.578700 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-webhook-cert\") pod \"maas-controller-579cf7c6d-f9vsn\" (UID: \"278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902\") " pod="opendatahub/maas-controller-579cf7c6d-f9vsn" Oct 31 03:22:37.578780 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.578755 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-metrics-certs\") pod \"maas-controller-579cf7c6d-f9vsn\" (UID: \"278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902\") " pod="opendatahub/maas-controller-579cf7c6d-f9vsn" Oct 31 03:22:37.679735 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.679700 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-metrics-certs\") pod \"maas-controller-579cf7c6d-f9vsn\" (UID: \"278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902\") " pod="opendatahub/maas-controller-579cf7c6d-f9vsn" Oct 31 03:22:37.679930 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.679772 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8wfs\" (UniqueName: \"kubernetes.io/projected/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-kube-api-access-p8wfs\") pod \"maas-controller-579cf7c6d-f9vsn\" (UID: \"278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902\") " pod="opendatahub/maas-controller-579cf7c6d-f9vsn" Oct 31 03:22:37.679994 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.679927 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-webhook-cert\") pod \"maas-controller-579cf7c6d-f9vsn\" (UID: \"278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902\") " pod="opendatahub/maas-controller-579cf7c6d-f9vsn" Oct 31 03:22:37.681482 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.681445 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:37.682581 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.682555 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-webhook-cert\") pod \"maas-controller-579cf7c6d-f9vsn\" (UID: \"278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902\") " pod="opendatahub/maas-controller-579cf7c6d-f9vsn" Oct 31 03:22:37.682663 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.682555 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-metrics-certs\") pod \"maas-controller-579cf7c6d-f9vsn\" (UID: \"278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902\") " pod="opendatahub/maas-controller-579cf7c6d-f9vsn" Oct 31 03:22:37.687944 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.687921 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8wfs\" (UniqueName: \"kubernetes.io/projected/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-kube-api-access-p8wfs\") pod \"maas-controller-579cf7c6d-f9vsn\" (UID: \"278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902\") " pod="opendatahub/maas-controller-579cf7c6d-f9vsn" Oct 31 03:22:37.823218 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.823125 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-f9vsn" Oct 31 03:22:37.833546 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.833514 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:37.834083 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.834065 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-586ddd746f-nv8gm"] Oct 31 03:22:37.837584 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.837566 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586ddd746f-nv8gm" Oct 31 03:22:37.854530 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.854498 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-586ddd746f-nv8gm"] Oct 31 03:22:37.986472 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.986434 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2238dc08-3a3e-426e-a8f7-22bec5f016c8-webhook-cert\") pod \"maas-controller-586ddd746f-nv8gm\" (UID: \"2238dc08-3a3e-426e-a8f7-22bec5f016c8\") " pod="opendatahub/maas-controller-586ddd746f-nv8gm" Oct 31 03:22:37.986658 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.986509 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk745\" (UniqueName: \"kubernetes.io/projected/2238dc08-3a3e-426e-a8f7-22bec5f016c8-kube-api-access-bk745\") pod \"maas-controller-586ddd746f-nv8gm\" (UID: \"2238dc08-3a3e-426e-a8f7-22bec5f016c8\") " pod="opendatahub/maas-controller-586ddd746f-nv8gm" Oct 31 03:22:37.986658 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:37.986552 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2238dc08-3a3e-426e-a8f7-22bec5f016c8-metrics-certs\") pod \"maas-controller-586ddd746f-nv8gm\" (UID: \"2238dc08-3a3e-426e-a8f7-22bec5f016c8\") " pod="opendatahub/maas-controller-586ddd746f-nv8gm" Oct 31 03:22:38.087178 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:38.087088 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk745\" (UniqueName: \"kubernetes.io/projected/2238dc08-3a3e-426e-a8f7-22bec5f016c8-kube-api-access-bk745\") pod \"maas-controller-586ddd746f-nv8gm\" (UID: \"2238dc08-3a3e-426e-a8f7-22bec5f016c8\") " pod="opendatahub/maas-controller-586ddd746f-nv8gm" Oct 31 03:22:38.087178 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:38.087142 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2238dc08-3a3e-426e-a8f7-22bec5f016c8-metrics-certs\") pod \"maas-controller-586ddd746f-nv8gm\" (UID: \"2238dc08-3a3e-426e-a8f7-22bec5f016c8\") " pod="opendatahub/maas-controller-586ddd746f-nv8gm" Oct 31 03:22:38.087178 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:38.087170 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2238dc08-3a3e-426e-a8f7-22bec5f016c8-webhook-cert\") pod \"maas-controller-586ddd746f-nv8gm\" (UID: \"2238dc08-3a3e-426e-a8f7-22bec5f016c8\") " pod="opendatahub/maas-controller-586ddd746f-nv8gm" Oct 31 03:22:38.089565 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:38.089539 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2238dc08-3a3e-426e-a8f7-22bec5f016c8-metrics-certs\") pod \"maas-controller-586ddd746f-nv8gm\" (UID: \"2238dc08-3a3e-426e-a8f7-22bec5f016c8\") " pod="opendatahub/maas-controller-586ddd746f-nv8gm" Oct 31 03:22:38.089660 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:38.089576 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2238dc08-3a3e-426e-a8f7-22bec5f016c8-webhook-cert\") pod \"maas-controller-586ddd746f-nv8gm\" (UID: \"2238dc08-3a3e-426e-a8f7-22bec5f016c8\") " pod="opendatahub/maas-controller-586ddd746f-nv8gm" Oct 31 03:22:38.094647 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:38.094623 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk745\" (UniqueName: \"kubernetes.io/projected/2238dc08-3a3e-426e-a8f7-22bec5f016c8-kube-api-access-bk745\") pod \"maas-controller-586ddd746f-nv8gm\" (UID: \"2238dc08-3a3e-426e-a8f7-22bec5f016c8\") " pod="opendatahub/maas-controller-586ddd746f-nv8gm" Oct 31 03:22:38.099696 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:38.099657 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-f9vsn"] Oct 31 03:22:38.101795 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:22:38.101770 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278c21da_9fb8_4fb3_a6f8_f8a7fd6bc902.slice/crio-a8e6ed4be51cc1f8c814474af667176c2c21f9038d9a04b766f61a664ad21048 WatchSource:0}: Error finding container a8e6ed4be51cc1f8c814474af667176c2c21f9038d9a04b766f61a664ad21048: Status 404 returned error can't find the container with id a8e6ed4be51cc1f8c814474af667176c2c21f9038d9a04b766f61a664ad21048 Oct 31 03:22:38.159427 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:38.159388 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586ddd746f-nv8gm" Oct 31 03:22:38.242205 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:38.242169 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:38.364930 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:38.364843 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:38.387079 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:38.387046 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-f9vsn" event={"ID":"278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902","Type":"ContainerStarted","Data":"a8e6ed4be51cc1f8c814474af667176c2c21f9038d9a04b766f61a664ad21048"} Oct 31 03:22:38.431588 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:38.431563 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-586ddd746f-nv8gm"] Oct 31 03:22:38.433481 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:22:38.433453 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2238dc08_3a3e_426e_a8f7_22bec5f016c8.slice/crio-2f67fe9e6635eef884229986ab8bb74fe3eabb37e01e3a9657e6e4b202125138 WatchSource:0}: Error finding container 2f67fe9e6635eef884229986ab8bb74fe3eabb37e01e3a9657e6e4b202125138: Status 404 returned error can't find the container with id 2f67fe9e6635eef884229986ab8bb74fe3eabb37e01e3a9657e6e4b202125138 Oct 31 03:22:39.400062 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:39.400016 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586ddd746f-nv8gm" event={"ID":"2238dc08-3a3e-426e-a8f7-22bec5f016c8","Type":"ContainerStarted","Data":"2f67fe9e6635eef884229986ab8bb74fe3eabb37e01e3a9657e6e4b202125138"} Oct 31 03:22:41.415349 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:41.415306 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-f9vsn" event={"ID":"278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902","Type":"ContainerStarted","Data":"c5e2dd21d390c98f460ef3519140cec2b6cf470c6e8b99997803cd62382fc595"} Oct 31 03:22:41.415807 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:41.415380 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-f9vsn" Oct 31 03:22:41.417070 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:41.417045 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586ddd746f-nv8gm" event={"ID":"2238dc08-3a3e-426e-a8f7-22bec5f016c8","Type":"ContainerStarted","Data":"090b98d525cb749c2a0dbb244befddc00cca4ea40d6a6a013699eb4035ae57d8"} Oct 31 03:22:41.417192 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:41.417177 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-586ddd746f-nv8gm" Oct 31 03:22:41.485188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:41.485134 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-f9vsn" podStartSLOduration=1.46834589 podStartE2EDuration="4.485121689s" totalImagesPullingTime="3.016775799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:41.484114504 +0000 UTC m=+396.018809459" watchObservedRunningTime="2026-10-31 03:22:41.485121689 +0000 UTC m=+396.019816645" Oct 31 03:22:41.512535 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:41.512494 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:41.523239 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:41.523185 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-586ddd746f-nv8gm" podStartSLOduration=1.634053905 podStartE2EDuration="4.523172261s" totalImagesPullingTime="2.889118356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:41.521111567 +0000 UTC m=+396.055806521" watchObservedRunningTime="2026-10-31 03:22:41.523172261 +0000 UTC m=+396.057867258" Oct 31 03:22:41.803238 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:41.803192 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:43.063275 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.063236 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb"] Oct 31 03:22:43.067565 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.067535 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:22:43.074868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.074575 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 31 03:22:43.074868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.074592 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 31 03:22:43.074868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.074641 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 31 03:22:43.074868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.074826 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-567jj\"" Oct 31 03:22:43.080848 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.080824 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb"] Oct 31 03:22:43.185552 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.185513 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:43.236604 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.236560 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/55f29950-78b4-4a52-818d-dc235e4631fc-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68drb\" (UID: \"55f29950-78b4-4a52-818d-dc235e4631fc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:22:43.236824 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.236610 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss42p\" (UniqueName: \"kubernetes.io/projected/55f29950-78b4-4a52-818d-dc235e4631fc-kube-api-access-ss42p\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68drb\" (UID: \"55f29950-78b4-4a52-818d-dc235e4631fc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:22:43.236824 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.236778 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55f29950-78b4-4a52-818d-dc235e4631fc-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68drb\" (UID: \"55f29950-78b4-4a52-818d-dc235e4631fc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:22:43.236824 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.236811 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/55f29950-78b4-4a52-818d-dc235e4631fc-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68drb\" (UID: \"55f29950-78b4-4a52-818d-dc235e4631fc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:22:43.264499 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.264464 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" Oct 31 03:22:43.326006 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.325910 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:43.337928 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.337900 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/55f29950-78b4-4a52-818d-dc235e4631fc-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68drb\" (UID: \"55f29950-78b4-4a52-818d-dc235e4631fc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:22:43.338076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.337966 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/55f29950-78b4-4a52-818d-dc235e4631fc-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68drb\" (UID: \"55f29950-78b4-4a52-818d-dc235e4631fc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:22:43.338076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.337998 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss42p\" (UniqueName: \"kubernetes.io/projected/55f29950-78b4-4a52-818d-dc235e4631fc-kube-api-access-ss42p\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68drb\" (UID: \"55f29950-78b4-4a52-818d-dc235e4631fc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:22:43.338076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.338049 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55f29950-78b4-4a52-818d-dc235e4631fc-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68drb\" (UID: \"55f29950-78b4-4a52-818d-dc235e4631fc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:22:43.338987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.338962 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55f29950-78b4-4a52-818d-dc235e4631fc-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68drb\" (UID: \"55f29950-78b4-4a52-818d-dc235e4631fc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:22:43.339433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.339410 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/55f29950-78b4-4a52-818d-dc235e4631fc-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68drb\" (UID: \"55f29950-78b4-4a52-818d-dc235e4631fc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:22:43.340453 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.340434 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/55f29950-78b4-4a52-818d-dc235e4631fc-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68drb\" (UID: \"55f29950-78b4-4a52-818d-dc235e4631fc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:22:43.346558 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.346533 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss42p\" (UniqueName: \"kubernetes.io/projected/55f29950-78b4-4a52-818d-dc235e4631fc-kube-api-access-ss42p\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-68drb\" (UID: \"55f29950-78b4-4a52-818d-dc235e4631fc\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:22:43.382861 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.382830 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:22:43.658434 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:43.658409 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb"] Oct 31 03:22:43.660577 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:22:43.660548 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f29950_78b4_4a52_818d_dc235e4631fc.slice/crio-7172e69a131816fd5169c25cdc306f8d44919a73f2f4e78124157635011b1adb WatchSource:0}: Error finding container 7172e69a131816fd5169c25cdc306f8d44919a73f2f4e78124157635011b1adb: Status 404 returned error can't find the container with id 7172e69a131816fd5169c25cdc306f8d44919a73f2f4e78124157635011b1adb Oct 31 03:22:44.432205 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:44.432165 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" event={"ID":"55f29950-78b4-4a52-818d-dc235e4631fc","Type":"ContainerStarted","Data":"7172e69a131816fd5169c25cdc306f8d44919a73f2f4e78124157635011b1adb"} Oct 31 03:22:47.137297 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:22:47.137192 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:22:52.424635 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:52.424601 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-586ddd746f-nv8gm" Oct 31 03:22:52.425476 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:52.425453 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-f9vsn" Oct 31 03:22:52.496795 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:52.496755 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-f9vsn"] Oct 31 03:22:52.497071 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:52.497021 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-f9vsn" podUID="278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902" containerName="manager" containerID="cri-o://c5e2dd21d390c98f460ef3519140cec2b6cf470c6e8b99997803cd62382fc595" gracePeriod=30 Oct 31 03:22:52.755406 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:52.755365 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-f9vsn" Oct 31 03:22:52.840564 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:52.840522 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-kube-api-access-p8wfs\" (UniqueName: \"kubernetes.io/projected/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-kube-api-access-p8wfs\") pod \"278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902\" (UID: \"278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902\") " Oct 31 03:22:52.840776 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:52.840623 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-webhook-cert\" (UniqueName: \"kubernetes.io/secret/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-webhook-cert\") pod \"278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902\" (UID: \"278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902\") " Oct 31 03:22:52.840776 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:52.840709 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-metrics-certs\" (UniqueName: \"kubernetes.io/secret/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-metrics-certs\") pod \"278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902\" (UID: \"278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902\") " Oct 31 03:22:52.843343 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:52.843304 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-webhook-cert" pod "278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902" (UID: "278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:52.843452 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:52.843365 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-kube-api-access-p8wfs" pod "278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902" (UID: "278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902"). InnerVolumeSpecName "kube-api-access-p8wfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:22:52.843452 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:52.843370 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-metrics-certs" pod "278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902" (UID: "278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:22:52.942239 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:52.942157 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:52.942239 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:52.942192 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8wfs\" (UniqueName: \"kubernetes.io/projected/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-kube-api-access-p8wfs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:52.942239 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:52.942207 5490 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902-webhook-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:22:53.489399 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:53.489358 5490 generic.go:356] "Generic (PLEG): container finished" podID="278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902" containerID="c5e2dd21d390c98f460ef3519140cec2b6cf470c6e8b99997803cd62382fc595" exitCode=0 Oct 31 03:22:53.489940 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:53.489416 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-f9vsn" Oct 31 03:22:53.489940 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:53.489438 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-f9vsn" event={"ID":"278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902","Type":"ContainerDied","Data":"c5e2dd21d390c98f460ef3519140cec2b6cf470c6e8b99997803cd62382fc595"} Oct 31 03:22:53.489940 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:53.489501 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-f9vsn" event={"ID":"278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902","Type":"ContainerDied","Data":"a8e6ed4be51cc1f8c814474af667176c2c21f9038d9a04b766f61a664ad21048"} Oct 31 03:22:53.489940 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:53.489532 5490 scope.go:122] "RemoveContainer" containerID="c5e2dd21d390c98f460ef3519140cec2b6cf470c6e8b99997803cd62382fc595" Oct 31 03:22:53.500806 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:53.500612 5490 scope.go:122] "RemoveContainer" containerID="c5e2dd21d390c98f460ef3519140cec2b6cf470c6e8b99997803cd62382fc595" Oct 31 03:22:53.501023 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:22:53.501003 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e2dd21d390c98f460ef3519140cec2b6cf470c6e8b99997803cd62382fc595\": container with ID starting with c5e2dd21d390c98f460ef3519140cec2b6cf470c6e8b99997803cd62382fc595 not found: ID does not exist" containerID="c5e2dd21d390c98f460ef3519140cec2b6cf470c6e8b99997803cd62382fc595" Oct 31 03:22:53.501076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:53.501035 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e2dd21d390c98f460ef3519140cec2b6cf470c6e8b99997803cd62382fc595"} err="failed to get container status \"c5e2dd21d390c98f460ef3519140cec2b6cf470c6e8b99997803cd62382fc595\": rpc error: code = NotFound desc = could not find container \"c5e2dd21d390c98f460ef3519140cec2b6cf470c6e8b99997803cd62382fc595\": container with ID starting with c5e2dd21d390c98f460ef3519140cec2b6cf470c6e8b99997803cd62382fc595 not found: ID does not exist" Oct 31 03:22:53.513196 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:53.513163 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-f9vsn"] Oct 31 03:22:53.516255 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:53.516233 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-f9vsn"] Oct 31 03:22:53.574909 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:53.574872 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902" path="/var/lib/kubelet/pods/278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902/volumes" Oct 31 03:22:55.526026 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:55.525988 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:22:56.507333 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:56.507295 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" event={"ID":"55f29950-78b4-4a52-818d-dc235e4631fc","Type":"ContainerStarted","Data":"20c1a8fe58f1b857d42ccc0528bd0a4c43f1aa5bbcec5f42f3409e3ac599573f"} Oct 31 03:22:56.507522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:56.507391 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:22:56.532874 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:22:56.532819 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" podStartSLOduration=1.417734184 podStartE2EDuration="13.532803485s" totalImagesPullingTime="12.115069301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:56.53153887 +0000 UTC m=+411.066233826" watchObservedRunningTime="2026-10-31 03:22:56.532803485 +0000 UTC m=+411.067498440" Oct 31 03:23:07.518571 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:07.518537 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:23:12.785291 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:12.785227 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:12.933695 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:12.933636 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:14.097040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.097001 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk"] Oct 31 03:23:14.097474 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.097456 5490 cpu_manager.go:414] "containerMap: removing container" podUID="278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902" containerName="manager" Oct 31 03:23:14.097516 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.097476 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902" containerName="manager" Oct 31 03:23:14.097553 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.097545 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="278c21da-9fb8-4fb3-a6f8-f8a7fd6bc902" containerName="manager" Oct 31 03:23:14.104912 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.104876 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:14.107503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.107483 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" Oct 31 03:23:14.114785 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.114757 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 03:23:14.114929 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.114816 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 03:23:14.120072 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.120046 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk"] Oct 31 03:23:14.238289 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.238256 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-metrics-certs\") pod \"maas-api-6d669c7fcd-6xmnk\" (UID: \"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" Oct 31 03:23:14.238472 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.238312 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jms9w\" (UniqueName: \"kubernetes.io/projected/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-kube-api-access-jms9w\") pod \"maas-api-6d669c7fcd-6xmnk\" (UID: \"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" Oct 31 03:23:14.238472 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.238413 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-maas-api-tls\") pod \"maas-api-6d669c7fcd-6xmnk\" (UID: \"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" Oct 31 03:23:14.238472 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.238447 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-openshift-service-ca\") pod \"maas-api-6d669c7fcd-6xmnk\" (UID: \"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" Oct 31 03:23:14.338844 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.338808 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:14.339091 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.339069 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-maas-api-tls\") pod \"maas-api-6d669c7fcd-6xmnk\" (UID: \"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" Oct 31 03:23:14.339160 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.339107 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-openshift-service-ca\") pod \"maas-api-6d669c7fcd-6xmnk\" (UID: \"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" Oct 31 03:23:14.339160 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.339151 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-metrics-certs\") pod \"maas-api-6d669c7fcd-6xmnk\" (UID: \"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" Oct 31 03:23:14.339225 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.339195 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jms9w\" (UniqueName: \"kubernetes.io/projected/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-kube-api-access-jms9w\") pod \"maas-api-6d669c7fcd-6xmnk\" (UID: \"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" Oct 31 03:23:14.339919 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.339894 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-openshift-service-ca\") pod \"maas-api-6d669c7fcd-6xmnk\" (UID: \"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" Oct 31 03:23:14.341639 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.341620 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-metrics-certs\") pod \"maas-api-6d669c7fcd-6xmnk\" (UID: \"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" Oct 31 03:23:14.341748 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.341730 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-maas-api-tls\") pod \"maas-api-6d669c7fcd-6xmnk\" (UID: \"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" Oct 31 03:23:14.351744 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.351667 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jms9w\" (UniqueName: \"kubernetes.io/projected/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-kube-api-access-jms9w\") pod \"maas-api-6d669c7fcd-6xmnk\" (UID: \"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" Oct 31 03:23:14.424721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.424685 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" Oct 31 03:23:14.703810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:14.703769 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk"] Oct 31 03:23:14.706865 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:23:14.706820 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87a079e_75dd_420d_ac2f_cd6e2f8c7d57.slice/crio-d71f8c5fa8735aecea8c358b909d93a316d50076d367b2e50ab12f9f76b3c35b WatchSource:0}: Error finding container d71f8c5fa8735aecea8c358b909d93a316d50076d367b2e50ab12f9f76b3c35b: Status 404 returned error can't find the container with id d71f8c5fa8735aecea8c358b909d93a316d50076d367b2e50ab12f9f76b3c35b Oct 31 03:23:15.290637 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:15.290601 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:15.604474 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:15.604385 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" event={"ID":"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57","Type":"ContainerStarted","Data":"d71f8c5fa8735aecea8c358b909d93a316d50076d367b2e50ab12f9f76b3c35b"} Oct 31 03:23:16.611169 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:16.611126 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" event={"ID":"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57","Type":"ContainerStarted","Data":"52c11ad6bc23748c52047f663167960378601631f1811cc5bc8f2e8a3a13a980"} Oct 31 03:23:16.611604 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:16.611259 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" Oct 31 03:23:17.140950 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:23:17.140901 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:23:22.621633 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:22.621597 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" Oct 31 03:23:22.642290 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:22.642230 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" podStartSLOduration=6.8748383749999995 podStartE2EDuration="8.642215702s" totalImagesPullingTime="1.767377327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:16.641979758 +0000 UTC m=+431.176674713" watchObservedRunningTime="2026-10-31 03:23:22.642215702 +0000 UTC m=+437.176910682" Oct 31 03:23:24.081412 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.081374 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:24.237503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.237466 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:23:24.354280 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.354206 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-p67sx"] Oct 31 03:23:24.600930 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.600893 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-p67sx"] Oct 31 03:23:24.601102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.600979 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-p67sx"] Oct 31 03:23:24.601102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.601004 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh"] Oct 31 03:23:24.601102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.600998 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-p67sx" Oct 31 03:23:24.601422 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:23:24.601398 5490 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-t4t9w], unattached volumes=[], failed to process volumes=[kube-api-access-t4t9w]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-p67sx" podUID="b9a0dd9a-b491-442d-9745-5e2e7c20e037" Oct 31 03:23:24.629387 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.629317 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh"] Oct 31 03:23:24.629525 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.629431 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" Oct 31 03:23:24.632972 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.632953 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dbwgh\"" Oct 31 03:23:24.673407 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.673377 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-p67sx" Oct 31 03:23:24.694268 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.694239 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-p67sx" Oct 31 03:23:24.723113 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.723079 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b46bbb97d-xfzck"] Oct 31 03:23:24.740838 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.740804 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mntm7\" (UniqueName: \"kubernetes.io/projected/6f4d14ca-ec36-4daa-8971-93c7dd709b48-kube-api-access-mntm7\") pod \"maas-api-78fdcdcc4-gp5mh\" (UID: \"6f4d14ca-ec36-4daa-8971-93c7dd709b48\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" Oct 31 03:23:24.740838 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.740852 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4d14ca-ec36-4daa-8971-93c7dd709b48-openshift-service-ca\") pod \"maas-api-78fdcdcc4-gp5mh\" (UID: \"6f4d14ca-ec36-4daa-8971-93c7dd709b48\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" Oct 31 03:23:24.741084 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.740875 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4d14ca-ec36-4daa-8971-93c7dd709b48-maas-api-tls\") pod \"maas-api-78fdcdcc4-gp5mh\" (UID: \"6f4d14ca-ec36-4daa-8971-93c7dd709b48\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" Oct 31 03:23:24.741084 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.741006 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f4d14ca-ec36-4daa-8971-93c7dd709b48-metrics-certs\") pod \"maas-api-78fdcdcc4-gp5mh\" (UID: \"6f4d14ca-ec36-4daa-8971-93c7dd709b48\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" Oct 31 03:23:24.748705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.748667 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b46bbb97d-xfzck"] Oct 31 03:23:24.748838 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.748782 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b46bbb97d-xfzck" Oct 31 03:23:24.752687 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.752648 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b2rkt\"" Oct 31 03:23:24.753080 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.753053 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 03:23:24.842301 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.842265 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4d14ca-ec36-4daa-8971-93c7dd709b48-openshift-service-ca\") pod \"maas-api-78fdcdcc4-gp5mh\" (UID: \"6f4d14ca-ec36-4daa-8971-93c7dd709b48\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" Oct 31 03:23:24.923093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.842312 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4d14ca-ec36-4daa-8971-93c7dd709b48-maas-api-tls\") pod \"maas-api-78fdcdcc4-gp5mh\" (UID: \"6f4d14ca-ec36-4daa-8971-93c7dd709b48\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" Oct 31 03:23:24.923093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.842357 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52bfc\" (UniqueName: \"kubernetes.io/projected/fcafd3c6-e514-4b26-963b-e02b7032fdd2-kube-api-access-52bfc\") pod \"authorino-7b46bbb97d-xfzck\" (UID: \"fcafd3c6-e514-4b26-963b-e02b7032fdd2\") " pod="kuadrant-system/authorino-7b46bbb97d-xfzck" Oct 31 03:23:24.923093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.842389 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f4d14ca-ec36-4daa-8971-93c7dd709b48-metrics-certs\") pod \"maas-api-78fdcdcc4-gp5mh\" (UID: \"6f4d14ca-ec36-4daa-8971-93c7dd709b48\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" Oct 31 03:23:24.923093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.842450 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fcafd3c6-e514-4b26-963b-e02b7032fdd2-tls-cert\") pod \"authorino-7b46bbb97d-xfzck\" (UID: \"fcafd3c6-e514-4b26-963b-e02b7032fdd2\") " pod="kuadrant-system/authorino-7b46bbb97d-xfzck" Oct 31 03:23:24.923093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.842661 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mntm7\" (UniqueName: \"kubernetes.io/projected/6f4d14ca-ec36-4daa-8971-93c7dd709b48-kube-api-access-mntm7\") pod \"maas-api-78fdcdcc4-gp5mh\" (UID: \"6f4d14ca-ec36-4daa-8971-93c7dd709b48\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" Oct 31 03:23:24.923093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.843067 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4d14ca-ec36-4daa-8971-93c7dd709b48-openshift-service-ca\") pod \"maas-api-78fdcdcc4-gp5mh\" (UID: \"6f4d14ca-ec36-4daa-8971-93c7dd709b48\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" Oct 31 03:23:24.923093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.844852 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4d14ca-ec36-4daa-8971-93c7dd709b48-maas-api-tls\") pod \"maas-api-78fdcdcc4-gp5mh\" (UID: \"6f4d14ca-ec36-4daa-8971-93c7dd709b48\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" Oct 31 03:23:24.923093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.845236 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f4d14ca-ec36-4daa-8971-93c7dd709b48-metrics-certs\") pod \"maas-api-78fdcdcc4-gp5mh\" (UID: \"6f4d14ca-ec36-4daa-8971-93c7dd709b48\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" Oct 31 03:23:24.923093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.850354 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mntm7\" (UniqueName: \"kubernetes.io/projected/6f4d14ca-ec36-4daa-8971-93c7dd709b48-kube-api-access-mntm7\") pod \"maas-api-78fdcdcc4-gp5mh\" (UID: \"6f4d14ca-ec36-4daa-8971-93c7dd709b48\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" Oct 31 03:23:24.943299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.943267 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52bfc\" (UniqueName: \"kubernetes.io/projected/fcafd3c6-e514-4b26-963b-e02b7032fdd2-kube-api-access-52bfc\") pod \"authorino-7b46bbb97d-xfzck\" (UID: \"fcafd3c6-e514-4b26-963b-e02b7032fdd2\") " pod="kuadrant-system/authorino-7b46bbb97d-xfzck" Oct 31 03:23:24.943434 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.943335 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fcafd3c6-e514-4b26-963b-e02b7032fdd2-tls-cert\") pod \"authorino-7b46bbb97d-xfzck\" (UID: \"fcafd3c6-e514-4b26-963b-e02b7032fdd2\") " pod="kuadrant-system/authorino-7b46bbb97d-xfzck" Oct 31 03:23:24.945617 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.945596 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fcafd3c6-e514-4b26-963b-e02b7032fdd2-tls-cert\") pod \"authorino-7b46bbb97d-xfzck\" (UID: \"fcafd3c6-e514-4b26-963b-e02b7032fdd2\") " pod="kuadrant-system/authorino-7b46bbb97d-xfzck" Oct 31 03:23:24.951926 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.951903 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52bfc\" (UniqueName: \"kubernetes.io/projected/fcafd3c6-e514-4b26-963b-e02b7032fdd2-kube-api-access-52bfc\") pod \"authorino-7b46bbb97d-xfzck\" (UID: \"fcafd3c6-e514-4b26-963b-e02b7032fdd2\") " pod="kuadrant-system/authorino-7b46bbb97d-xfzck" Oct 31 03:23:24.978933 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:24.978899 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" Oct 31 03:23:25.088181 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:25.088098 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b46bbb97d-xfzck" Oct 31 03:23:25.255625 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:25.255600 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh"] Oct 31 03:23:25.257830 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:23:25.257801 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4d14ca_ec36_4daa_8971_93c7dd709b48.slice/crio-89cfcf2928ace145b676860f6394b9fae824c4cc7e87e514d6a478382d8f3fd7 WatchSource:0}: Error finding container 89cfcf2928ace145b676860f6394b9fae824c4cc7e87e514d6a478382d8f3fd7: Status 404 returned error can't find the container with id 89cfcf2928ace145b676860f6394b9fae824c4cc7e87e514d6a478382d8f3fd7 Oct 31 03:23:25.376189 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:25.376164 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b46bbb97d-xfzck"] Oct 31 03:23:25.378632 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:23:25.378606 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcafd3c6_e514_4b26_963b_e02b7032fdd2.slice/crio-d1f735e4b7667cd01810f94d27f42604bee0608345b387c7f748f7be0509f0b2 WatchSource:0}: Error finding container d1f735e4b7667cd01810f94d27f42604bee0608345b387c7f748f7be0509f0b2: Status 404 returned error can't find the container with id d1f735e4b7667cd01810f94d27f42604bee0608345b387c7f748f7be0509f0b2 Oct 31 03:23:25.679759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:25.679720 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b46bbb97d-xfzck" event={"ID":"fcafd3c6-e514-4b26-963b-e02b7032fdd2","Type":"ContainerStarted","Data":"d1f735e4b7667cd01810f94d27f42604bee0608345b387c7f748f7be0509f0b2"} Oct 31 03:23:25.681102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:25.681070 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" event={"ID":"6f4d14ca-ec36-4daa-8971-93c7dd709b48","Type":"ContainerStarted","Data":"89cfcf2928ace145b676860f6394b9fae824c4cc7e87e514d6a478382d8f3fd7"} Oct 31 03:23:25.682635 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:25.682611 5490 generic.go:356] "Generic (PLEG): container finished" podID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" containerID="5095b4b424c27e0d1b4508340a2ef081f59d900607606f5547cd9d9696dd72b7" exitCode=0 Oct 31 03:23:25.682758 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:25.682695 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" event={"ID":"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666","Type":"ContainerDied","Data":"5095b4b424c27e0d1b4508340a2ef081f59d900607606f5547cd9d9696dd72b7"} Oct 31 03:23:25.682812 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:25.682704 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-p67sx" Oct 31 03:23:25.683176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:25.683162 5490 scope.go:122] "RemoveContainer" containerID="5095b4b424c27e0d1b4508340a2ef081f59d900607606f5547cd9d9696dd72b7" Oct 31 03:23:25.684268 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:25.684234 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:23:25.684372 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:25.684308 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:23:25.684372 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:25.684349 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:23:25.739982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:25.739951 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-p67sx"] Oct 31 03:23:25.746604 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:25.746576 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-p67sx"] Oct 31 03:23:26.688879 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:26.688834 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" event={"ID":"6f4d14ca-ec36-4daa-8971-93c7dd709b48","Type":"ContainerStarted","Data":"b012e17c9dc41fe2d34e8a61df9f0477d53fac64981e794e80574219cd79ecc7"} Oct 31 03:23:26.689369 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:26.688901 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" Oct 31 03:23:26.690863 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:26.690840 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" event={"ID":"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666","Type":"ContainerStarted","Data":"042b2b5699dc8a0e8f1b9b4fc6fa574f84cbd93097f524eac8d7af8f87354bb6"} Oct 31 03:23:26.705808 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:26.705760 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" podStartSLOduration=1.9140161070000001 podStartE2EDuration="2.705746328s" totalImagesPullingTime="791.730221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:26.704006677 +0000 UTC m=+441.238701632" watchObservedRunningTime="2026-10-31 03:23:26.705746328 +0000 UTC m=+441.240441283" Oct 31 03:23:27.198650 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:27.198612 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:23:27.199367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:27.199345 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-k9bcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body= Oct 31 03:23:27.199425 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:27.199403 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" podUID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" Oct 31 03:23:27.579735 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:27.579652 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a0dd9a-b491-442d-9745-5e2e7c20e037" path="/var/lib/kubelet/pods/b9a0dd9a-b491-442d-9745-5e2e7c20e037/volumes" Oct 31 03:23:27.698123 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:27.698075 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b46bbb97d-xfzck" event={"ID":"fcafd3c6-e514-4b26-963b-e02b7032fdd2","Type":"ContainerStarted","Data":"a6bc68f21bc8a5e0f3d43957dbbd29421c17acb82cfdf6f2db3d29a6959d8333"} Oct 31 03:23:27.714487 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:27.714435 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7b46bbb97d-xfzck" podStartSLOduration=2.378320702 podStartE2EDuration="3.714419926s" totalImagesPullingTime="1.336099224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:27.712794326 +0000 UTC m=+442.247489281" watchObservedRunningTime="2026-10-31 03:23:27.714419926 +0000 UTC m=+442.249114880" Oct 31 03:23:28.198379 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:28.198336 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-k9bcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body= Oct 31 03:23:28.198574 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:28.198400 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" podUID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" Oct 31 03:23:29.198036 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:29.197993 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-k9bcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body= Oct 31 03:23:29.198426 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:29.198052 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" podUID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" Oct 31 03:23:30.198063 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:30.198026 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-k9bcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body= Oct 31 03:23:30.198485 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:30.198081 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" podUID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" Oct 31 03:23:31.198563 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:31.198523 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-k9bcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" start-of-body= Oct 31 03:23:31.198993 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:31.198583 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" podUID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": dial tcp 10.133.0.21:15021: connect: connection refused" Oct 31 03:23:32.202194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:32.202160 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:23:32.202582 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:32.202454 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:23:32.203227 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:32.203210 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:23:32.703797 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:32.703767 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" Oct 31 03:23:32.755968 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:32.755931 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk"] Oct 31 03:23:32.756265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:32.756214 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" podUID="b87a079e-75dd-420d-ac2f-cd6e2f8c7d57" containerName="maas-api" containerID="cri-o://52c11ad6bc23748c52047f663167960378601631f1811cc5bc8f2e8a3a13a980" gracePeriod=30 Oct 31 03:23:32.999171 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:32.999135 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" Oct 31 03:23:33.132714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.132646 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-maas-api-tls\") pod \"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57\" (UID: \"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57\") " Oct 31 03:23:33.132890 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.132787 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-metrics-certs\") pod \"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57\" (UID: \"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57\") " Oct 31 03:23:33.132890 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.132846 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-openshift-service-ca\") pod \"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57\" (UID: \"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57\") " Oct 31 03:23:33.132968 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.132893 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-kube-api-access-jms9w\" (UniqueName: \"kubernetes.io/projected/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-kube-api-access-jms9w\") pod \"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57\" (UID: \"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57\") " Oct 31 03:23:33.133262 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.133222 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-openshift-service-ca" pod "b87a079e-75dd-420d-ac2f-cd6e2f8c7d57" (UID: "b87a079e-75dd-420d-ac2f-cd6e2f8c7d57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:23:33.134923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.134887 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-maas-api-tls" pod "b87a079e-75dd-420d-ac2f-cd6e2f8c7d57" (UID: "b87a079e-75dd-420d-ac2f-cd6e2f8c7d57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:23:33.135037 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.134990 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-kube-api-access-jms9w" pod "b87a079e-75dd-420d-ac2f-cd6e2f8c7d57" (UID: "b87a079e-75dd-420d-ac2f-cd6e2f8c7d57"). InnerVolumeSpecName "kube-api-access-jms9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:23:33.135037 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.134997 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-metrics-certs" pod "b87a079e-75dd-420d-ac2f-cd6e2f8c7d57" (UID: "b87a079e-75dd-420d-ac2f-cd6e2f8c7d57"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:23:33.234457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.234361 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:33.234457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.234402 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:33.234457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.234418 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jms9w\" (UniqueName: \"kubernetes.io/projected/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-kube-api-access-jms9w\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:33.234457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.234431 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:23:33.730109 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.730066 5490 generic.go:356] "Generic (PLEG): container finished" podID="b87a079e-75dd-420d-ac2f-cd6e2f8c7d57" containerID="52c11ad6bc23748c52047f663167960378601631f1811cc5bc8f2e8a3a13a980" exitCode=0 Oct 31 03:23:33.730293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.730132 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" Oct 31 03:23:33.730293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.730152 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" event={"ID":"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57","Type":"ContainerDied","Data":"52c11ad6bc23748c52047f663167960378601631f1811cc5bc8f2e8a3a13a980"} Oct 31 03:23:33.730293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.730207 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk" event={"ID":"b87a079e-75dd-420d-ac2f-cd6e2f8c7d57","Type":"ContainerDied","Data":"d71f8c5fa8735aecea8c358b909d93a316d50076d367b2e50ab12f9f76b3c35b"} Oct 31 03:23:33.730293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.730229 5490 scope.go:122] "RemoveContainer" containerID="52c11ad6bc23748c52047f663167960378601631f1811cc5bc8f2e8a3a13a980" Oct 31 03:23:33.739377 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.739357 5490 scope.go:122] "RemoveContainer" containerID="52c11ad6bc23748c52047f663167960378601631f1811cc5bc8f2e8a3a13a980" Oct 31 03:23:33.739687 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:23:33.739650 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c11ad6bc23748c52047f663167960378601631f1811cc5bc8f2e8a3a13a980\": container with ID starting with 52c11ad6bc23748c52047f663167960378601631f1811cc5bc8f2e8a3a13a980 not found: ID does not exist" containerID="52c11ad6bc23748c52047f663167960378601631f1811cc5bc8f2e8a3a13a980" Oct 31 03:23:33.739827 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.739804 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c11ad6bc23748c52047f663167960378601631f1811cc5bc8f2e8a3a13a980"} err="failed to get container status \"52c11ad6bc23748c52047f663167960378601631f1811cc5bc8f2e8a3a13a980\": rpc error: code = NotFound desc = could not find container \"52c11ad6bc23748c52047f663167960378601631f1811cc5bc8f2e8a3a13a980\": container with ID starting with 52c11ad6bc23748c52047f663167960378601631f1811cc5bc8f2e8a3a13a980 not found: ID does not exist" Oct 31 03:23:33.748528 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.748501 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk"] Oct 31 03:23:33.750627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:33.750604 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-6xmnk"] Oct 31 03:23:35.574312 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:23:35.574282 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b87a079e-75dd-420d-ac2f-cd6e2f8c7d57" path="/var/lib/kubelet/pods/b87a079e-75dd-420d-ac2f-cd6e2f8c7d57/volumes" Oct 31 03:23:47.126833 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:23:47.126739 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:24:02.556922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:02.556877 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:02.703509 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:02.703474 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:15.327624 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:15.327576 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:17.126326 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:24:17.126288 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:24:20.871155 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:20.871115 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:22.075117 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.075081 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-vczbq"] Oct 31 03:24:22.075501 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.075486 5490 cpu_manager.go:414] "containerMap: removing container" podUID="b87a079e-75dd-420d-ac2f-cd6e2f8c7d57" containerName="maas-api" Oct 31 03:24:22.075543 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.075502 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b87a079e-75dd-420d-ac2f-cd6e2f8c7d57" containerName="maas-api" Oct 31 03:24:22.075603 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.075594 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="b87a079e-75dd-420d-ac2f-cd6e2f8c7d57" containerName="maas-api" Oct 31 03:24:22.078938 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.078916 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:22.094063 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.094016 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tkb4s\"" Oct 31 03:24:22.094063 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.094026 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 31 03:24:22.096614 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.096583 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-vczbq"] Oct 31 03:24:22.128468 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.128423 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9ecffa6-8de1-421d-952e-a9eeecf8d862-home\") pod \"echo-simulator-kserve-64b6f8f96c-vczbq\" (UID: \"e9ecffa6-8de1-421d-952e-a9eeecf8d862\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:22.128713 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.128506 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrqmr\" (UniqueName: \"kubernetes.io/projected/e9ecffa6-8de1-421d-952e-a9eeecf8d862-kube-api-access-wrqmr\") pod \"echo-simulator-kserve-64b6f8f96c-vczbq\" (UID: \"e9ecffa6-8de1-421d-952e-a9eeecf8d862\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:22.128713 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.128607 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9ecffa6-8de1-421d-952e-a9eeecf8d862-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-vczbq\" (UID: \"e9ecffa6-8de1-421d-952e-a9eeecf8d862\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:22.128843 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.128729 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9ecffa6-8de1-421d-952e-a9eeecf8d862-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-vczbq\" (UID: \"e9ecffa6-8de1-421d-952e-a9eeecf8d862\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:22.128843 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.128769 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9ecffa6-8de1-421d-952e-a9eeecf8d862-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-vczbq\" (UID: \"e9ecffa6-8de1-421d-952e-a9eeecf8d862\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:22.128843 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.128813 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9ecffa6-8de1-421d-952e-a9eeecf8d862-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-vczbq\" (UID: \"e9ecffa6-8de1-421d-952e-a9eeecf8d862\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:22.146878 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.146841 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 03:24:22.146878 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.146862 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 03:24:22.177799 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.177747 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:22.229413 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.229376 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9ecffa6-8de1-421d-952e-a9eeecf8d862-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-vczbq\" (UID: \"e9ecffa6-8de1-421d-952e-a9eeecf8d862\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:22.229413 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.229415 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9ecffa6-8de1-421d-952e-a9eeecf8d862-home\") pod \"echo-simulator-kserve-64b6f8f96c-vczbq\" (UID: \"e9ecffa6-8de1-421d-952e-a9eeecf8d862\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:22.229654 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.229459 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrqmr\" (UniqueName: \"kubernetes.io/projected/e9ecffa6-8de1-421d-952e-a9eeecf8d862-kube-api-access-wrqmr\") pod \"echo-simulator-kserve-64b6f8f96c-vczbq\" (UID: \"e9ecffa6-8de1-421d-952e-a9eeecf8d862\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:22.229654 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.229488 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9ecffa6-8de1-421d-952e-a9eeecf8d862-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-vczbq\" (UID: \"e9ecffa6-8de1-421d-952e-a9eeecf8d862\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:22.229654 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.229555 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9ecffa6-8de1-421d-952e-a9eeecf8d862-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-vczbq\" (UID: \"e9ecffa6-8de1-421d-952e-a9eeecf8d862\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:22.229654 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.229576 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9ecffa6-8de1-421d-952e-a9eeecf8d862-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-vczbq\" (UID: \"e9ecffa6-8de1-421d-952e-a9eeecf8d862\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:22.229991 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.229968 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9ecffa6-8de1-421d-952e-a9eeecf8d862-home\") pod \"echo-simulator-kserve-64b6f8f96c-vczbq\" (UID: \"e9ecffa6-8de1-421d-952e-a9eeecf8d862\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:22.230047 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.230023 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9ecffa6-8de1-421d-952e-a9eeecf8d862-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-vczbq\" (UID: \"e9ecffa6-8de1-421d-952e-a9eeecf8d862\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:22.230096 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.230057 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9ecffa6-8de1-421d-952e-a9eeecf8d862-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-vczbq\" (UID: \"e9ecffa6-8de1-421d-952e-a9eeecf8d862\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:22.231857 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.231833 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9ecffa6-8de1-421d-952e-a9eeecf8d862-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-vczbq\" (UID: \"e9ecffa6-8de1-421d-952e-a9eeecf8d862\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:22.232054 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.232039 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9ecffa6-8de1-421d-952e-a9eeecf8d862-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-vczbq\" (UID: \"e9ecffa6-8de1-421d-952e-a9eeecf8d862\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:22.237014 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.236994 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrqmr\" (UniqueName: \"kubernetes.io/projected/e9ecffa6-8de1-421d-952e-a9eeecf8d862-kube-api-access-wrqmr\") pod \"echo-simulator-kserve-64b6f8f96c-vczbq\" (UID: \"e9ecffa6-8de1-421d-952e-a9eeecf8d862\") " pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:22.393197 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.393105 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:22.669108 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.669073 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-vczbq"] Oct 31 03:24:22.874486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.874436 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm"] Oct 31 03:24:22.880352 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.880324 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:22.888313 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.888278 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 31 03:24:22.890639 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.890607 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm"] Oct 31 03:24:22.946896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.946810 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f64d3e84-440a-4298-9230-ae9e0063c962-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm\" (UID: \"f64d3e84-440a-4298-9230-ae9e0063c962\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:22.946896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.946851 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f64d3e84-440a-4298-9230-ae9e0063c962-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm\" (UID: \"f64d3e84-440a-4298-9230-ae9e0063c962\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:22.946896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.946873 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f64d3e84-440a-4298-9230-ae9e0063c962-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm\" (UID: \"f64d3e84-440a-4298-9230-ae9e0063c962\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:22.947123 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.946952 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb5dt\" (UniqueName: \"kubernetes.io/projected/f64d3e84-440a-4298-9230-ae9e0063c962-kube-api-access-jb5dt\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm\" (UID: \"f64d3e84-440a-4298-9230-ae9e0063c962\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:22.947123 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.947039 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f64d3e84-440a-4298-9230-ae9e0063c962-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm\" (UID: \"f64d3e84-440a-4298-9230-ae9e0063c962\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:22.947123 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.947063 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f64d3e84-440a-4298-9230-ae9e0063c962-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm\" (UID: \"f64d3e84-440a-4298-9230-ae9e0063c962\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:22.969121 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.969083 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" event={"ID":"e9ecffa6-8de1-421d-952e-a9eeecf8d862","Type":"ContainerStarted","Data":"5565135546dbc663042c30607cbd4c5e80b8935180b13a731a801793ba22a539"} Oct 31 03:24:22.977361 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:22.977328 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:23.048008 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:23.047976 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f64d3e84-440a-4298-9230-ae9e0063c962-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm\" (UID: \"f64d3e84-440a-4298-9230-ae9e0063c962\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:23.048239 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:23.048021 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f64d3e84-440a-4298-9230-ae9e0063c962-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm\" (UID: \"f64d3e84-440a-4298-9230-ae9e0063c962\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:23.048393 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:23.048365 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f64d3e84-440a-4298-9230-ae9e0063c962-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm\" (UID: \"f64d3e84-440a-4298-9230-ae9e0063c962\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:23.048458 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:23.048428 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb5dt\" (UniqueName: \"kubernetes.io/projected/f64d3e84-440a-4298-9230-ae9e0063c962-kube-api-access-jb5dt\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm\" (UID: \"f64d3e84-440a-4298-9230-ae9e0063c962\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:23.048516 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:23.048478 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f64d3e84-440a-4298-9230-ae9e0063c962-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm\" (UID: \"f64d3e84-440a-4298-9230-ae9e0063c962\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:23.048516 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:23.048512 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f64d3e84-440a-4298-9230-ae9e0063c962-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm\" (UID: \"f64d3e84-440a-4298-9230-ae9e0063c962\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:23.048742 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:23.048711 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f64d3e84-440a-4298-9230-ae9e0063c962-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm\" (UID: \"f64d3e84-440a-4298-9230-ae9e0063c962\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:23.049096 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:23.049070 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f64d3e84-440a-4298-9230-ae9e0063c962-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm\" (UID: \"f64d3e84-440a-4298-9230-ae9e0063c962\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:23.049186 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:23.049146 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f64d3e84-440a-4298-9230-ae9e0063c962-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm\" (UID: \"f64d3e84-440a-4298-9230-ae9e0063c962\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:23.051368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:23.051324 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f64d3e84-440a-4298-9230-ae9e0063c962-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm\" (UID: \"f64d3e84-440a-4298-9230-ae9e0063c962\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:23.051830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:23.051809 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f64d3e84-440a-4298-9230-ae9e0063c962-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm\" (UID: \"f64d3e84-440a-4298-9230-ae9e0063c962\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:23.056505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:23.056479 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb5dt\" (UniqueName: \"kubernetes.io/projected/f64d3e84-440a-4298-9230-ae9e0063c962-kube-api-access-jb5dt\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm\" (UID: \"f64d3e84-440a-4298-9230-ae9e0063c962\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:23.201232 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:23.201130 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:23.499241 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:23.499202 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm"] Oct 31 03:24:23.504510 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:24:23.504480 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64d3e84_440a_4298_9230_ae9e0063c962.slice/crio-ff8f741c3f82e05e4b6325451ddd3828ee75bd89554b1371289eeac5ac31ee2a WatchSource:0}: Error finding container ff8f741c3f82e05e4b6325451ddd3828ee75bd89554b1371289eeac5ac31ee2a: Status 404 returned error can't find the container with id ff8f741c3f82e05e4b6325451ddd3828ee75bd89554b1371289eeac5ac31ee2a Oct 31 03:24:23.978115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:23.978068 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" event={"ID":"f64d3e84-440a-4298-9230-ae9e0063c962","Type":"ContainerStarted","Data":"ff8f741c3f82e05e4b6325451ddd3828ee75bd89554b1371289eeac5ac31ee2a"} Oct 31 03:24:24.150633 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:24.150594 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:25.997441 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:25.997404 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" event={"ID":"f64d3e84-440a-4298-9230-ae9e0063c962","Type":"ContainerStarted","Data":"59b857a2101cd2e75bdb227c81948eafb37894e3eb91348f22064d9be19f280e"} Oct 31 03:24:25.999096 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:25.999067 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" event={"ID":"e9ecffa6-8de1-421d-952e-a9eeecf8d862","Type":"ContainerStarted","Data":"58d219a1a9cdf95540abb158f6eaadab8cae0e1a519c4a127f0201877f879c43"} Oct 31 03:24:26.039669 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.039595 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podStartSLOduration=1.606552333 podStartE2EDuration="4.039574549s" totalImagesPullingTime="2.433022216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:26.037638696 +0000 UTC m=+500.572333679" watchObservedRunningTime="2026-10-31 03:24:26.039574549 +0000 UTC m=+500.574269506" Oct 31 03:24:26.040221 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.040180 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" podStartSLOduration=2.133600381 podStartE2EDuration="4.04017086s" totalImagesPullingTime="1.906570479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:26.017487495 +0000 UTC m=+500.552182451" watchObservedRunningTime="2026-10-31 03:24:26.04017086 +0000 UTC m=+500.574865819" Oct 31 03:24:26.201528 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.201494 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:26.393997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.393900 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:26.501223 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.501193 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8"] Oct 31 03:24:26.505354 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.505331 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:26.513587 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.513562 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 31 03:24:26.516716 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.516670 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8"] Oct 31 03:24:26.519314 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.519288 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8ml7\" (UniqueName: \"kubernetes.io/projected/775a8940-cb22-4774-a43f-c6e7c5364354-kube-api-access-l8ml7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8\" (UID: \"775a8940-cb22-4774-a43f-c6e7c5364354\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:26.519440 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.519337 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/775a8940-cb22-4774-a43f-c6e7c5364354-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8\" (UID: \"775a8940-cb22-4774-a43f-c6e7c5364354\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:26.519506 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.519457 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/775a8940-cb22-4774-a43f-c6e7c5364354-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8\" (UID: \"775a8940-cb22-4774-a43f-c6e7c5364354\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:26.519562 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.519534 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/775a8940-cb22-4774-a43f-c6e7c5364354-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8\" (UID: \"775a8940-cb22-4774-a43f-c6e7c5364354\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:26.519637 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.519611 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775a8940-cb22-4774-a43f-c6e7c5364354-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8\" (UID: \"775a8940-cb22-4774-a43f-c6e7c5364354\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:26.519735 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.519722 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/775a8940-cb22-4774-a43f-c6e7c5364354-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8\" (UID: \"775a8940-cb22-4774-a43f-c6e7c5364354\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:26.598616 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.598571 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:26.620791 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.620754 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775a8940-cb22-4774-a43f-c6e7c5364354-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8\" (UID: \"775a8940-cb22-4774-a43f-c6e7c5364354\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:26.620945 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.620820 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/775a8940-cb22-4774-a43f-c6e7c5364354-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8\" (UID: \"775a8940-cb22-4774-a43f-c6e7c5364354\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:26.620991 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.620975 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8ml7\" (UniqueName: \"kubernetes.io/projected/775a8940-cb22-4774-a43f-c6e7c5364354-kube-api-access-l8ml7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8\" (UID: \"775a8940-cb22-4774-a43f-c6e7c5364354\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:26.621043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.621025 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/775a8940-cb22-4774-a43f-c6e7c5364354-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8\" (UID: \"775a8940-cb22-4774-a43f-c6e7c5364354\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:26.621085 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.621070 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/775a8940-cb22-4774-a43f-c6e7c5364354-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8\" (UID: \"775a8940-cb22-4774-a43f-c6e7c5364354\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:26.621136 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.621120 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/775a8940-cb22-4774-a43f-c6e7c5364354-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8\" (UID: \"775a8940-cb22-4774-a43f-c6e7c5364354\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:26.621227 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.621206 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/775a8940-cb22-4774-a43f-c6e7c5364354-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8\" (UID: \"775a8940-cb22-4774-a43f-c6e7c5364354\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:26.621429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.621403 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/775a8940-cb22-4774-a43f-c6e7c5364354-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8\" (UID: \"775a8940-cb22-4774-a43f-c6e7c5364354\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:26.621530 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.621449 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/775a8940-cb22-4774-a43f-c6e7c5364354-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8\" (UID: \"775a8940-cb22-4774-a43f-c6e7c5364354\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:26.623399 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.623377 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/775a8940-cb22-4774-a43f-c6e7c5364354-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8\" (UID: \"775a8940-cb22-4774-a43f-c6e7c5364354\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:26.623471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.623421 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/775a8940-cb22-4774-a43f-c6e7c5364354-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8\" (UID: \"775a8940-cb22-4774-a43f-c6e7c5364354\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:26.633824 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.633796 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8ml7\" (UniqueName: \"kubernetes.io/projected/775a8940-cb22-4774-a43f-c6e7c5364354-kube-api-access-l8ml7\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8\" (UID: \"775a8940-cb22-4774-a43f-c6e7c5364354\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:26.821663 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:26.821624 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:27.096261 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:27.096235 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8"] Oct 31 03:24:27.098091 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:24:27.098065 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775a8940_cb22_4774_a43f_c6e7c5364354.slice/crio-7503cdcb0162c4cf041aa335c10221d95070333a09823f0167f30c06a66513f7 WatchSource:0}: Error finding container 7503cdcb0162c4cf041aa335c10221d95070333a09823f0167f30c06a66513f7: Status 404 returned error can't find the container with id 7503cdcb0162c4cf041aa335c10221d95070333a09823f0167f30c06a66513f7 Oct 31 03:24:28.014714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:28.013767 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" event={"ID":"775a8940-cb22-4774-a43f-c6e7c5364354","Type":"ContainerStarted","Data":"f04a345e498fd8ce330bee0f4d97daca597b75596a4d46202f5a8fa90a76df4b"} Oct 31 03:24:28.014714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:28.013825 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" event={"ID":"775a8940-cb22-4774-a43f-c6e7c5364354","Type":"ContainerStarted","Data":"7503cdcb0162c4cf041aa335c10221d95070333a09823f0167f30c06a66513f7"} Oct 31 03:24:28.032567 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:28.032515 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" podStartSLOduration=1.706915109 podStartE2EDuration="2.032496285s" totalImagesPullingTime="325.581176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:28.030615564 +0000 UTC m=+502.565310521" watchObservedRunningTime="2026-10-31 03:24:28.032496285 +0000 UTC m=+502.567191239" Oct 31 03:24:28.449294 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:28.449251 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:28.822273 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:28.822185 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:29.792658 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:29.792622 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:30.689612 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.689571 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf"] Oct 31 03:24:30.694304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.694273 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:30.702601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.702055 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 31 03:24:30.703621 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.703595 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf"] Oct 31 03:24:30.761931 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.761884 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80180652-fb4f-4976-956d-25a47344d581-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-b78nf\" (UID: \"80180652-fb4f-4976-956d-25a47344d581\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:30.762150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.762026 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80180652-fb4f-4976-956d-25a47344d581-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-b78nf\" (UID: \"80180652-fb4f-4976-956d-25a47344d581\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:30.762150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.762069 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80180652-fb4f-4976-956d-25a47344d581-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-b78nf\" (UID: \"80180652-fb4f-4976-956d-25a47344d581\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:30.762150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.762123 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp6xc\" (UniqueName: \"kubernetes.io/projected/80180652-fb4f-4976-956d-25a47344d581-kube-api-access-rp6xc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-b78nf\" (UID: \"80180652-fb4f-4976-956d-25a47344d581\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:30.762267 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.762239 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80180652-fb4f-4976-956d-25a47344d581-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-b78nf\" (UID: \"80180652-fb4f-4976-956d-25a47344d581\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:30.762379 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.762358 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80180652-fb4f-4976-956d-25a47344d581-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-b78nf\" (UID: \"80180652-fb4f-4976-956d-25a47344d581\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:30.779283 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.779243 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:30.863952 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.863909 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp6xc\" (UniqueName: \"kubernetes.io/projected/80180652-fb4f-4976-956d-25a47344d581-kube-api-access-rp6xc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-b78nf\" (UID: \"80180652-fb4f-4976-956d-25a47344d581\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:30.864437 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.863984 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80180652-fb4f-4976-956d-25a47344d581-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-b78nf\" (UID: \"80180652-fb4f-4976-956d-25a47344d581\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:30.864437 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.864245 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80180652-fb4f-4976-956d-25a47344d581-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-b78nf\" (UID: \"80180652-fb4f-4976-956d-25a47344d581\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:30.864437 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.864373 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80180652-fb4f-4976-956d-25a47344d581-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-b78nf\" (UID: \"80180652-fb4f-4976-956d-25a47344d581\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:30.864621 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.864444 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80180652-fb4f-4976-956d-25a47344d581-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-b78nf\" (UID: \"80180652-fb4f-4976-956d-25a47344d581\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:30.864621 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.864473 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80180652-fb4f-4976-956d-25a47344d581-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-b78nf\" (UID: \"80180652-fb4f-4976-956d-25a47344d581\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:30.864780 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.864696 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80180652-fb4f-4976-956d-25a47344d581-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-b78nf\" (UID: \"80180652-fb4f-4976-956d-25a47344d581\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:30.864896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.864869 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80180652-fb4f-4976-956d-25a47344d581-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-b78nf\" (UID: \"80180652-fb4f-4976-956d-25a47344d581\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:30.865167 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.865144 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80180652-fb4f-4976-956d-25a47344d581-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-b78nf\" (UID: \"80180652-fb4f-4976-956d-25a47344d581\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:30.867020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.866974 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80180652-fb4f-4976-956d-25a47344d581-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-b78nf\" (UID: \"80180652-fb4f-4976-956d-25a47344d581\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:30.867178 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.867153 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80180652-fb4f-4976-956d-25a47344d581-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-b78nf\" (UID: \"80180652-fb4f-4976-956d-25a47344d581\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:30.872050 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:30.872022 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp6xc\" (UniqueName: \"kubernetes.io/projected/80180652-fb4f-4976-956d-25a47344d581-kube-api-access-rp6xc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-b78nf\" (UID: \"80180652-fb4f-4976-956d-25a47344d581\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:31.011736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:31.011611 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:31.295995 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:31.295966 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf"] Oct 31 03:24:31.298327 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:24:31.298298 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80180652_fb4f_4976_956d_25a47344d581.slice/crio-34c925f99d4b2664acb16cd712069c846fca58d33cced52f35e68be7baec62c3 WatchSource:0}: Error finding container 34c925f99d4b2664acb16cd712069c846fca58d33cced52f35e68be7baec62c3: Status 404 returned error can't find the container with id 34c925f99d4b2664acb16cd712069c846fca58d33cced52f35e68be7baec62c3 Oct 31 03:24:32.040973 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:32.040882 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" event={"ID":"80180652-fb4f-4976-956d-25a47344d581","Type":"ContainerStarted","Data":"675ad1d0e1f1c35c653611e67f09fec05c546af06b79623a1c062a13ec35b33b"} Oct 31 03:24:32.040973 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:32.040942 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" event={"ID":"80180652-fb4f-4976-956d-25a47344d581","Type":"ContainerStarted","Data":"34c925f99d4b2664acb16cd712069c846fca58d33cced52f35e68be7baec62c3"} Oct 31 03:24:32.062861 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:32.062811 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" podStartSLOduration=1.704892857 podStartE2EDuration="2.062796494s" totalImagesPullingTime="357.903637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:32.060609259 +0000 UTC m=+506.595304215" watchObservedRunningTime="2026-10-31 03:24:32.062796494 +0000 UTC m=+506.597491450" Oct 31 03:24:32.393484 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:32.393386 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:32.407042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:32.407013 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:33.012002 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:33.011968 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:33.060655 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:33.060613 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:24:33.201655 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:33.201610 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:33.214231 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:33.214194 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:34.064210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:34.064176 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:24:36.822382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:36.822334 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:36.835032 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:36.835004 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:37.078391 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:37.078312 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:24:41.012784 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.012746 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:41.026282 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.026253 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:41.099977 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.099950 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:24:41.514111 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.514076 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr"] Oct 31 03:24:41.517840 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.517816 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.532336 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.532299 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr"] Oct 31 03:24:41.572030 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.571999 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.572195 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.572062 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.572195 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.572144 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.572195 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.572181 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.572375 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.572255 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.572375 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.572301 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.572375 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.572323 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.572524 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.572405 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.572524 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.572425 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6scn\" (UniqueName: \"kubernetes.io/projected/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-kube-api-access-j6scn\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.673698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.673637 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.673904 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.673712 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6scn\" (UniqueName: \"kubernetes.io/projected/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-kube-api-access-j6scn\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.673904 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.673749 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.673904 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.673829 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.673904 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.673863 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.674119 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.673905 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.674119 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.673951 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.674119 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.673976 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.674119 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.674002 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.674379 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.674249 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.674543 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.674519 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.674875 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.674608 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.675054 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.674882 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.675054 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.674951 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.676429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.676411 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.676949 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.676919 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.684180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.684155 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.684879 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.684860 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6scn\" (UniqueName: \"kubernetes.io/projected/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-kube-api-access-j6scn\") pod \"maas-default-gateway-openshift-default-769df7846c-kkbfr\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:41.831864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:41.831778 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:42.118621 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:42.118592 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr"] Oct 31 03:24:42.121216 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:24:42.121183 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680c32cf_d69b_4d52_b696_ca7ecb54e0e9.slice/crio-f16987c8943eef53ffa5ee7bdf8ccf7163ade2100cc1f03db99c6fb8bc75c448 WatchSource:0}: Error finding container f16987c8943eef53ffa5ee7bdf8ccf7163ade2100cc1f03db99c6fb8bc75c448: Status 404 returned error can't find the container with id f16987c8943eef53ffa5ee7bdf8ccf7163ade2100cc1f03db99c6fb8bc75c448 Oct 31 03:24:42.123359 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:42.123331 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:24:42.123439 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:42.123390 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:24:42.123439 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:42.123420 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:24:42.852332 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:42.841783 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b8d5bdb45-vldp7"] Oct 31 03:24:42.854779 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:42.854738 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b8d5bdb45-vldp7"] Oct 31 03:24:42.854950 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:42.854861 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b8d5bdb45-vldp7" Oct 31 03:24:42.986593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:42.986554 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hmtz\" (UniqueName: \"kubernetes.io/projected/c8be91e5-ce63-477c-91c3-4a24e694a0f3-kube-api-access-6hmtz\") pod \"authorino-5b8d5bdb45-vldp7\" (UID: \"c8be91e5-ce63-477c-91c3-4a24e694a0f3\") " pod="kuadrant-system/authorino-5b8d5bdb45-vldp7" Oct 31 03:24:42.986866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:42.986617 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c8be91e5-ce63-477c-91c3-4a24e694a0f3-tls-cert\") pod \"authorino-5b8d5bdb45-vldp7\" (UID: \"c8be91e5-ce63-477c-91c3-4a24e694a0f3\") " pod="kuadrant-system/authorino-5b8d5bdb45-vldp7" Oct 31 03:24:43.087920 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:43.087882 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hmtz\" (UniqueName: \"kubernetes.io/projected/c8be91e5-ce63-477c-91c3-4a24e694a0f3-kube-api-access-6hmtz\") pod \"authorino-5b8d5bdb45-vldp7\" (UID: \"c8be91e5-ce63-477c-91c3-4a24e694a0f3\") " pod="kuadrant-system/authorino-5b8d5bdb45-vldp7" Oct 31 03:24:43.088089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:43.087945 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c8be91e5-ce63-477c-91c3-4a24e694a0f3-tls-cert\") pod \"authorino-5b8d5bdb45-vldp7\" (UID: \"c8be91e5-ce63-477c-91c3-4a24e694a0f3\") " pod="kuadrant-system/authorino-5b8d5bdb45-vldp7" Oct 31 03:24:43.090783 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:43.090757 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c8be91e5-ce63-477c-91c3-4a24e694a0f3-tls-cert\") pod \"authorino-5b8d5bdb45-vldp7\" (UID: \"c8be91e5-ce63-477c-91c3-4a24e694a0f3\") " pod="kuadrant-system/authorino-5b8d5bdb45-vldp7" Oct 31 03:24:43.096737 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:43.096716 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hmtz\" (UniqueName: \"kubernetes.io/projected/c8be91e5-ce63-477c-91c3-4a24e694a0f3-kube-api-access-6hmtz\") pod \"authorino-5b8d5bdb45-vldp7\" (UID: \"c8be91e5-ce63-477c-91c3-4a24e694a0f3\") " pod="kuadrant-system/authorino-5b8d5bdb45-vldp7" Oct 31 03:24:43.102938 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:43.102878 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" event={"ID":"680c32cf-d69b-4d52-b696-ca7ecb54e0e9","Type":"ContainerStarted","Data":"044b1407d9788c38a9ff0fc65a7457b0d542c8e952da7c306fe5bf7627d094df"} Oct 31 03:24:43.103031 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:43.102934 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" event={"ID":"680c32cf-d69b-4d52-b696-ca7ecb54e0e9","Type":"ContainerStarted","Data":"f16987c8943eef53ffa5ee7bdf8ccf7163ade2100cc1f03db99c6fb8bc75c448"} Oct 31 03:24:43.124228 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:43.124176 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" podStartSLOduration=2.124159963 podStartE2EDuration="2.124159963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:43.122120531 +0000 UTC m=+517.656815506" watchObservedRunningTime="2026-10-31 03:24:43.124159963 +0000 UTC m=+517.658854917" Oct 31 03:24:43.169171 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:43.169130 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b8d5bdb45-vldp7" Oct 31 03:24:43.456765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:43.456729 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b8d5bdb45-vldp7"] Oct 31 03:24:43.460102 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:24:43.460077 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8be91e5_ce63_477c_91c3_4a24e694a0f3.slice/crio-7ae932db1e9f0ef40b9b2a8bd20ac598172410db09d81bae425adcdb3b5e4b13 WatchSource:0}: Error finding container 7ae932db1e9f0ef40b9b2a8bd20ac598172410db09d81bae425adcdb3b5e4b13: Status 404 returned error can't find the container with id 7ae932db1e9f0ef40b9b2a8bd20ac598172410db09d81bae425adcdb3b5e4b13 Oct 31 03:24:43.848873 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:43.848363 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:43.848873 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:43.848496 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kkbfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 31 03:24:43.848873 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:43.848537 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" podUID="680c32cf-d69b-4d52-b696-ca7ecb54e0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 31 03:24:44.108920 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:44.108833 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b8d5bdb45-vldp7" event={"ID":"c8be91e5-ce63-477c-91c3-4a24e694a0f3","Type":"ContainerStarted","Data":"7ae932db1e9f0ef40b9b2a8bd20ac598172410db09d81bae425adcdb3b5e4b13"} Oct 31 03:24:44.836966 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:44.833127 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kkbfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 31 03:24:44.836966 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:44.833194 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" podUID="680c32cf-d69b-4d52-b696-ca7ecb54e0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 31 03:24:45.119360 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.119271 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b8d5bdb45-vldp7" event={"ID":"c8be91e5-ce63-477c-91c3-4a24e694a0f3","Type":"ContainerStarted","Data":"d5c48676379a4519586b42c993833be243aa5c99f37b9b064a8168a0b141b77a"} Oct 31 03:24:45.170986 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.170934 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5b8d5bdb45-vldp7" podStartSLOduration=2.338389315 podStartE2EDuration="3.170919198s" totalImagesPullingTime="832.529883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:45.16957868 +0000 UTC m=+519.704273635" watchObservedRunningTime="2026-10-31 03:24:45.170919198 +0000 UTC m=+519.705614153" Oct 31 03:24:45.253242 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.248865 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b46bbb97d-xfzck"] Oct 31 03:24:45.253242 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.251956 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7b46bbb97d-xfzck" podUID="fcafd3c6-e514-4b26-963b-e02b7032fdd2" containerName="authorino" containerID="cri-o://a6bc68f21bc8a5e0f3d43957dbbd29421c17acb82cfdf6f2db3d29a6959d8333" gracePeriod=30 Oct 31 03:24:45.580607 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.580578 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b46bbb97d-xfzck" Oct 31 03:24:45.622263 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.617048 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcafd3c6-e514-4b26-963b-e02b7032fdd2-kube-api-access-52bfc\" (UniqueName: \"kubernetes.io/projected/fcafd3c6-e514-4b26-963b-e02b7032fdd2-kube-api-access-52bfc\") pod \"fcafd3c6-e514-4b26-963b-e02b7032fdd2\" (UID: \"fcafd3c6-e514-4b26-963b-e02b7032fdd2\") " Oct 31 03:24:45.622263 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.617137 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcafd3c6-e514-4b26-963b-e02b7032fdd2-tls-cert\" (UniqueName: \"kubernetes.io/secret/fcafd3c6-e514-4b26-963b-e02b7032fdd2-tls-cert\") pod \"fcafd3c6-e514-4b26-963b-e02b7032fdd2\" (UID: \"fcafd3c6-e514-4b26-963b-e02b7032fdd2\") " Oct 31 03:24:45.623202 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.623151 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcafd3c6-e514-4b26-963b-e02b7032fdd2-kube-api-access-52bfc" pod "fcafd3c6-e514-4b26-963b-e02b7032fdd2" (UID: "fcafd3c6-e514-4b26-963b-e02b7032fdd2"). InnerVolumeSpecName "kube-api-access-52bfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:24:45.663293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.663257 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcafd3c6-e514-4b26-963b-e02b7032fdd2-tls-cert" pod "fcafd3c6-e514-4b26-963b-e02b7032fdd2" (UID: "fcafd3c6-e514-4b26-963b-e02b7032fdd2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:24:45.718326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.718296 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52bfc\" (UniqueName: \"kubernetes.io/projected/fcafd3c6-e514-4b26-963b-e02b7032fdd2-kube-api-access-52bfc\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:45.718326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.718324 5490 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fcafd3c6-e514-4b26-963b-e02b7032fdd2-tls-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:24:45.802138 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.802105 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj"] Oct 31 03:24:45.802628 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.802609 5490 cpu_manager.go:414] "containerMap: removing container" podUID="fcafd3c6-e514-4b26-963b-e02b7032fdd2" containerName="authorino" Oct 31 03:24:45.802628 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.802630 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcafd3c6-e514-4b26-963b-e02b7032fdd2" containerName="authorino" Oct 31 03:24:45.802821 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.802753 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcafd3c6-e514-4b26-963b-e02b7032fdd2" containerName="authorino" Oct 31 03:24:45.806023 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.806001 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj" Oct 31 03:24:45.812961 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.812940 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-4rf7f\"" Oct 31 03:24:45.820652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.819748 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 31 03:24:45.820652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.819795 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 31 03:24:45.821345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.821292 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj"] Oct 31 03:24:45.857901 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.857854 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kkbfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 31 03:24:45.858425 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.858075 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" podUID="680c32cf-d69b-4d52-b696-ca7ecb54e0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 31 03:24:45.859436 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.859410 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:45.921171 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.921113 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe17388f-bfb3-4ff1-841f-93d137ca9bc0-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-gpzfj\" (UID: \"fe17388f-bfb3-4ff1-841f-93d137ca9bc0\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj" Oct 31 03:24:45.921387 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:45.921323 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m9dq\" (UniqueName: \"kubernetes.io/projected/fe17388f-bfb3-4ff1-841f-93d137ca9bc0-kube-api-access-2m9dq\") pod \"ai-gateway-controller-5d9cd99f45-gpzfj\" (UID: \"fe17388f-bfb3-4ff1-841f-93d137ca9bc0\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj" Oct 31 03:24:46.007366 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:46.007322 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:46.022635 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:46.022601 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m9dq\" (UniqueName: \"kubernetes.io/projected/fe17388f-bfb3-4ff1-841f-93d137ca9bc0-kube-api-access-2m9dq\") pod \"ai-gateway-controller-5d9cd99f45-gpzfj\" (UID: \"fe17388f-bfb3-4ff1-841f-93d137ca9bc0\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj" Oct 31 03:24:46.022798 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:46.022655 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe17388f-bfb3-4ff1-841f-93d137ca9bc0-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-gpzfj\" (UID: \"fe17388f-bfb3-4ff1-841f-93d137ca9bc0\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj" Oct 31 03:24:46.025027 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:46.025003 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe17388f-bfb3-4ff1-841f-93d137ca9bc0-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-gpzfj\" (UID: \"fe17388f-bfb3-4ff1-841f-93d137ca9bc0\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj" Oct 31 03:24:46.054087 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:46.054049 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m9dq\" (UniqueName: \"kubernetes.io/projected/fe17388f-bfb3-4ff1-841f-93d137ca9bc0-kube-api-access-2m9dq\") pod \"ai-gateway-controller-5d9cd99f45-gpzfj\" (UID: \"fe17388f-bfb3-4ff1-841f-93d137ca9bc0\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj" Oct 31 03:24:46.122594 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:46.122451 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj" Oct 31 03:24:46.150700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:46.149947 5490 generic.go:356] "Generic (PLEG): container finished" podID="fcafd3c6-e514-4b26-963b-e02b7032fdd2" containerID="a6bc68f21bc8a5e0f3d43957dbbd29421c17acb82cfdf6f2db3d29a6959d8333" exitCode=0 Oct 31 03:24:46.150904 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:46.150861 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b46bbb97d-xfzck" Oct 31 03:24:46.152883 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:46.152847 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b46bbb97d-xfzck" event={"ID":"fcafd3c6-e514-4b26-963b-e02b7032fdd2","Type":"ContainerDied","Data":"a6bc68f21bc8a5e0f3d43957dbbd29421c17acb82cfdf6f2db3d29a6959d8333"} Oct 31 03:24:46.153092 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:46.153072 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b46bbb97d-xfzck" event={"ID":"fcafd3c6-e514-4b26-963b-e02b7032fdd2","Type":"ContainerDied","Data":"d1f735e4b7667cd01810f94d27f42604bee0608345b387c7f748f7be0509f0b2"} Oct 31 03:24:46.153213 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:46.153200 5490 scope.go:122] "RemoveContainer" containerID="a6bc68f21bc8a5e0f3d43957dbbd29421c17acb82cfdf6f2db3d29a6959d8333" Oct 31 03:24:46.167958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:46.167935 5490 scope.go:122] "RemoveContainer" containerID="a6bc68f21bc8a5e0f3d43957dbbd29421c17acb82cfdf6f2db3d29a6959d8333" Oct 31 03:24:46.168271 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:24:46.168244 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6bc68f21bc8a5e0f3d43957dbbd29421c17acb82cfdf6f2db3d29a6959d8333\": container with ID starting with a6bc68f21bc8a5e0f3d43957dbbd29421c17acb82cfdf6f2db3d29a6959d8333 not found: ID does not exist" containerID="a6bc68f21bc8a5e0f3d43957dbbd29421c17acb82cfdf6f2db3d29a6959d8333" Oct 31 03:24:46.168341 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:46.168309 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6bc68f21bc8a5e0f3d43957dbbd29421c17acb82cfdf6f2db3d29a6959d8333"} err="failed to get container status \"a6bc68f21bc8a5e0f3d43957dbbd29421c17acb82cfdf6f2db3d29a6959d8333\": rpc error: code = NotFound desc = could not find container \"a6bc68f21bc8a5e0f3d43957dbbd29421c17acb82cfdf6f2db3d29a6959d8333\": container with ID starting with a6bc68f21bc8a5e0f3d43957dbbd29421c17acb82cfdf6f2db3d29a6959d8333 not found: ID does not exist" Oct 31 03:24:46.176156 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:46.176095 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b46bbb97d-xfzck"] Oct 31 03:24:46.178918 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:46.178891 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7b46bbb97d-xfzck"] Oct 31 03:24:46.438515 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:46.438482 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj"] Oct 31 03:24:46.454295 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:24:46.454252 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe17388f_bfb3_4ff1_841f_93d137ca9bc0.slice/crio-26e3737eb34c64eb4e7bcd6f6a9d267fece89e5bf01e9a4f6c9b69658fa448e8 WatchSource:0}: Error finding container 26e3737eb34c64eb4e7bcd6f6a9d267fece89e5bf01e9a4f6c9b69658fa448e8: Status 404 returned error can't find the container with id 26e3737eb34c64eb4e7bcd6f6a9d267fece89e5bf01e9a4f6c9b69658fa448e8 Oct 31 03:24:46.837757 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:46.837058 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kkbfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 31 03:24:46.837757 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:46.837125 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" podUID="680c32cf-d69b-4d52-b696-ca7ecb54e0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 31 03:24:47.154901 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:24:47.154860 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:24:47.160369 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:47.160332 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj" event={"ID":"fe17388f-bfb3-4ff1-841f-93d137ca9bc0","Type":"ContainerStarted","Data":"26e3737eb34c64eb4e7bcd6f6a9d267fece89e5bf01e9a4f6c9b69658fa448e8"} Oct 31 03:24:47.574927 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:47.574843 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcafd3c6-e514-4b26-963b-e02b7032fdd2" path="/var/lib/kubelet/pods/fcafd3c6-e514-4b26-963b-e02b7032fdd2/volumes" Oct 31 03:24:47.841786 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:47.841199 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kkbfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 31 03:24:47.841786 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:47.841276 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" podUID="680c32cf-d69b-4d52-b696-ca7ecb54e0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 31 03:24:48.833101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:48.833050 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-kkbfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 31 03:24:48.833494 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:48.833142 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" podUID="680c32cf-d69b-4d52-b696-ca7ecb54e0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 31 03:24:50.713215 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:50.713182 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:50.713732 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:50.713574 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:50.714965 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:50.714932 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:24:51.189190 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:51.189151 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj" event={"ID":"fe17388f-bfb3-4ff1-841f-93d137ca9bc0","Type":"ContainerStarted","Data":"a8d9ddd0766a541617d38ced4d0e45b3e1557f51de7895768d6f9682f1e45cbf"} Oct 31 03:24:51.189490 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:51.189473 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj" Oct 31 03:24:51.214698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:51.211571 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj" podStartSLOduration=2.316078927 podStartE2EDuration="6.211551097s" totalImagesPullingTime="3.89547217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:51.209927259 +0000 UTC m=+525.744622214" watchObservedRunningTime="2026-10-31 03:24:51.211551097 +0000 UTC m=+525.746246053" Oct 31 03:24:52.271172 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.271134 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq"] Oct 31 03:24:52.275123 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.275098 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rclxh"] Oct 31 03:24:52.275271 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.275213 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" Oct 31 03:24:52.280644 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.279572 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" Oct 31 03:24:52.283373 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.283350 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 31 03:24:52.288606 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.288580 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq"] Oct 31 03:24:52.297338 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.297305 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rclxh"] Oct 31 03:24:52.297943 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.297909 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:52.422653 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.422609 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjr76\" (UniqueName: \"kubernetes.io/projected/c9d776b1-fabe-4396-adb1-030d1edad90c-kube-api-access-kjr76\") pod \"payload-pre-processing-7b98bb678c-gzjbq\" (UID: \"c9d776b1-fabe-4396-adb1-030d1edad90c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" Oct 31 03:24:52.422875 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.422768 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d776b1-fabe-4396-adb1-030d1edad90c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-gzjbq\" (UID: \"c9d776b1-fabe-4396-adb1-030d1edad90c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" Oct 31 03:24:52.422875 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.422802 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd6ql\" (UniqueName: \"kubernetes.io/projected/9e7f6e11-0382-4897-88e3-1d4283f5bed3-kube-api-access-bd6ql\") pod \"payload-processing-55fbbbf9dc-rclxh\" (UID: \"9e7f6e11-0382-4897-88e3-1d4283f5bed3\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" Oct 31 03:24:52.422875 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.422842 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7f6e11-0382-4897-88e3-1d4283f5bed3-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rclxh\" (UID: \"9e7f6e11-0382-4897-88e3-1d4283f5bed3\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" Oct 31 03:24:52.439396 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.439358 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:52.524028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.523921 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjr76\" (UniqueName: \"kubernetes.io/projected/c9d776b1-fabe-4396-adb1-030d1edad90c-kube-api-access-kjr76\") pod \"payload-pre-processing-7b98bb678c-gzjbq\" (UID: \"c9d776b1-fabe-4396-adb1-030d1edad90c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" Oct 31 03:24:52.524028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.523986 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d776b1-fabe-4396-adb1-030d1edad90c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-gzjbq\" (UID: \"c9d776b1-fabe-4396-adb1-030d1edad90c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" Oct 31 03:24:52.524028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.524015 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd6ql\" (UniqueName: \"kubernetes.io/projected/9e7f6e11-0382-4897-88e3-1d4283f5bed3-kube-api-access-bd6ql\") pod \"payload-processing-55fbbbf9dc-rclxh\" (UID: \"9e7f6e11-0382-4897-88e3-1d4283f5bed3\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" Oct 31 03:24:52.524326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.524068 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7f6e11-0382-4897-88e3-1d4283f5bed3-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rclxh\" (UID: \"9e7f6e11-0382-4897-88e3-1d4283f5bed3\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" Oct 31 03:24:52.524893 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.524869 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7f6e11-0382-4897-88e3-1d4283f5bed3-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-rclxh\" (UID: \"9e7f6e11-0382-4897-88e3-1d4283f5bed3\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" Oct 31 03:24:52.525022 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.524998 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d776b1-fabe-4396-adb1-030d1edad90c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-gzjbq\" (UID: \"c9d776b1-fabe-4396-adb1-030d1edad90c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" Oct 31 03:24:52.533380 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.533347 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjr76\" (UniqueName: \"kubernetes.io/projected/c9d776b1-fabe-4396-adb1-030d1edad90c-kube-api-access-kjr76\") pod \"payload-pre-processing-7b98bb678c-gzjbq\" (UID: \"c9d776b1-fabe-4396-adb1-030d1edad90c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" Oct 31 03:24:52.535107 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.535071 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd6ql\" (UniqueName: \"kubernetes.io/projected/9e7f6e11-0382-4897-88e3-1d4283f5bed3-kube-api-access-bd6ql\") pod \"payload-processing-55fbbbf9dc-rclxh\" (UID: \"9e7f6e11-0382-4897-88e3-1d4283f5bed3\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" Oct 31 03:24:52.592096 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.592039 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" Oct 31 03:24:52.603654 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.603623 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" Oct 31 03:24:52.883494 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.883464 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq"] Oct 31 03:24:52.884403 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:24:52.884376 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d776b1_fabe_4396_adb1_030d1edad90c.slice/crio-13e13aeff6e64744d7b6070caf3dda06c3b1b728d6095e21e39d3887313f46ae WatchSource:0}: Error finding container 13e13aeff6e64744d7b6070caf3dda06c3b1b728d6095e21e39d3887313f46ae: Status 404 returned error can't find the container with id 13e13aeff6e64744d7b6070caf3dda06c3b1b728d6095e21e39d3887313f46ae Oct 31 03:24:52.903372 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:52.903347 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-rclxh"] Oct 31 03:24:52.905752 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:24:52.905727 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7f6e11_0382_4897_88e3_1d4283f5bed3.slice/crio-180e5233156e9d7be16d452d41a08718591f25b47acb160d892704ebaacfb363 WatchSource:0}: Error finding container 180e5233156e9d7be16d452d41a08718591f25b47acb160d892704ebaacfb363: Status 404 returned error can't find the container with id 180e5233156e9d7be16d452d41a08718591f25b47acb160d892704ebaacfb363 Oct 31 03:24:53.201100 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:53.201062 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" event={"ID":"c9d776b1-fabe-4396-adb1-030d1edad90c","Type":"ContainerStarted","Data":"13e13aeff6e64744d7b6070caf3dda06c3b1b728d6095e21e39d3887313f46ae"} Oct 31 03:24:53.202519 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:53.202490 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" event={"ID":"9e7f6e11-0382-4897-88e3-1d4283f5bed3","Type":"ContainerStarted","Data":"180e5233156e9d7be16d452d41a08718591f25b47acb160d892704ebaacfb363"} Oct 31 03:24:56.220420 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.220374 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" event={"ID":"c9d776b1-fabe-4396-adb1-030d1edad90c","Type":"ContainerStarted","Data":"a25b8100ff503d22be8e8b108f183a1b631b010c79bb95430e3900ca25a2bf38"} Oct 31 03:24:56.220905 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.220473 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" Oct 31 03:24:56.221988 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.221962 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" event={"ID":"9e7f6e11-0382-4897-88e3-1d4283f5bed3","Type":"ContainerStarted","Data":"a003e8a3c8a5609ba2b52d72308648f30dabf4feefba87ee4feb9b1f237c32f9"} Oct 31 03:24:56.222127 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.222090 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" Oct 31 03:24:56.239528 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.239479 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" podStartSLOduration=1.4592381269999999 podStartE2EDuration="4.23946478s" totalImagesPullingTime="2.780226653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:56.237444539 +0000 UTC m=+530.772139495" watchObservedRunningTime="2026-10-31 03:24:56.23946478 +0000 UTC m=+530.774159735" Oct 31 03:24:56.255227 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.255177 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" podStartSLOduration=1.52978064 podStartE2EDuration="4.255162827s" totalImagesPullingTime="2.725382187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:56.253726223 +0000 UTC m=+530.788421181" watchObservedRunningTime="2026-10-31 03:24:56.255162827 +0000 UTC m=+530.789857782" Oct 31 03:24:56.615800 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.615710 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7"] Oct 31 03:24:56.619633 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.619611 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.640164 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.640134 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7"] Oct 31 03:24:56.797519 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.797484 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ffcf283-0d79-422e-82be-8b34db814efd-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.797747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.797530 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ffcf283-0d79-422e-82be-8b34db814efd-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.797747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.797549 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7dp7\" (UniqueName: \"kubernetes.io/projected/6ffcf283-0d79-422e-82be-8b34db814efd-kube-api-access-j7dp7\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.797747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.797701 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ffcf283-0d79-422e-82be-8b34db814efd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.797747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.797729 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ffcf283-0d79-422e-82be-8b34db814efd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.797982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.797755 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffcf283-0d79-422e-82be-8b34db814efd-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.797982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.797773 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ffcf283-0d79-422e-82be-8b34db814efd-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.797982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.797838 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffcf283-0d79-422e-82be-8b34db814efd-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.797982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.797903 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ffcf283-0d79-422e-82be-8b34db814efd-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.898864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.898783 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ffcf283-0d79-422e-82be-8b34db814efd-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.898864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.898853 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ffcf283-0d79-422e-82be-8b34db814efd-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.899047 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.898885 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ffcf283-0d79-422e-82be-8b34db814efd-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.899047 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.898908 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7dp7\" (UniqueName: \"kubernetes.io/projected/6ffcf283-0d79-422e-82be-8b34db814efd-kube-api-access-j7dp7\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.899047 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.898951 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ffcf283-0d79-422e-82be-8b34db814efd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.899047 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.898985 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ffcf283-0d79-422e-82be-8b34db814efd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.899047 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.899010 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffcf283-0d79-422e-82be-8b34db814efd-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.899047 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.899030 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ffcf283-0d79-422e-82be-8b34db814efd-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.899550 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.899055 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffcf283-0d79-422e-82be-8b34db814efd-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.899550 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.899283 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ffcf283-0d79-422e-82be-8b34db814efd-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.899550 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.899320 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ffcf283-0d79-422e-82be-8b34db814efd-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.899550 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.899387 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffcf283-0d79-422e-82be-8b34db814efd-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.899741 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.899606 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffcf283-0d79-422e-82be-8b34db814efd-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.899875 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.899857 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ffcf283-0d79-422e-82be-8b34db814efd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.901000 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.900971 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ffcf283-0d79-422e-82be-8b34db814efd-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.901475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.901453 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ffcf283-0d79-422e-82be-8b34db814efd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.907503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.907481 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ffcf283-0d79-422e-82be-8b34db814efd-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.907718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.907701 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7dp7\" (UniqueName: \"kubernetes.io/projected/6ffcf283-0d79-422e-82be-8b34db814efd-kube-api-access-j7dp7\") pod \"maas-default-gateway-openshift-default-769df7846c-t9bq7\" (UID: \"6ffcf283-0d79-422e-82be-8b34db814efd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:56.933647 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:56.933624 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:57.219374 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:57.219347 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7"] Oct 31 03:24:57.221196 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:24:57.221162 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ffcf283_0d79_422e_82be_8b34db814efd.slice/crio-6ad6b16384201c9b763b7ecd669911de6c82cf6b66a7ca5634cfe4a4180a0427 WatchSource:0}: Error finding container 6ad6b16384201c9b763b7ecd669911de6c82cf6b66a7ca5634cfe4a4180a0427: Status 404 returned error can't find the container with id 6ad6b16384201c9b763b7ecd669911de6c82cf6b66a7ca5634cfe4a4180a0427 Oct 31 03:24:57.223691 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:57.223648 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:24:57.223798 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:57.223731 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:24:57.223798 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:57.223761 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:24:57.228343 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:57.228313 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" event={"ID":"6ffcf283-0d79-422e-82be-8b34db814efd","Type":"ContainerStarted","Data":"6ad6b16384201c9b763b7ecd669911de6c82cf6b66a7ca5634cfe4a4180a0427"} Oct 31 03:24:57.701433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:57.701395 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:24:58.255703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:58.255648 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" event={"ID":"6ffcf283-0d79-422e-82be-8b34db814efd","Type":"ContainerStarted","Data":"6598e427d7f46b1336f05538f58529d9bf7bf4c2bb0386f20b07b27403ca07ee"} Oct 31 03:24:58.281834 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:58.281781 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" podStartSLOduration=2.281767504 podStartE2EDuration="2.281767504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:58.28022983 +0000 UTC m=+532.814924820" watchObservedRunningTime="2026-10-31 03:24:58.281767504 +0000 UTC m=+532.816462459" Oct 31 03:24:58.937064 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:58.937026 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:24:58.950701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:58.944488 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-t9bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 03:24:58.950701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:58.944548 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" podUID="6ffcf283-0d79-422e-82be-8b34db814efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 03:24:59.942162 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:59.942122 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-t9bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 03:24:59.942788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:24:59.942749 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" podUID="6ffcf283-0d79-422e-82be-8b34db814efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 03:25:00.938222 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:00.938176 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-t9bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 03:25:00.938540 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:00.938506 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" podUID="6ffcf283-0d79-422e-82be-8b34db814efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 03:25:01.936131 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:01.936090 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-t9bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 03:25:01.936720 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:01.936667 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" podUID="6ffcf283-0d79-422e-82be-8b34db814efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 03:25:02.198337 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:02.198260 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj" Oct 31 03:25:02.938703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:02.937129 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-t9bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 03:25:02.938703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:02.937208 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" podUID="6ffcf283-0d79-422e-82be-8b34db814efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 03:25:03.934663 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:03.934623 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-t9bq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 31 03:25:03.934901 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:03.934707 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" podUID="6ffcf283-0d79-422e-82be-8b34db814efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 31 03:25:05.535643 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:05.535605 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:25:05.536104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:05.536080 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:25:05.536881 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:05.536858 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" Oct 31 03:25:07.232139 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:07.232111 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" Oct 31 03:25:07.232522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:07.232176 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" Oct 31 03:25:11.615657 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.615622 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx"] Oct 31 03:25:11.624112 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.624088 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.630951 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.630851 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx"] Oct 31 03:25:11.659978 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.659936 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c5b443-e562-448f-a540-2273f5ae86ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.659978 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.659977 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c5b443-e562-448f-a540-2273f5ae86ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.660238 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.660012 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.660238 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.660123 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfcmm\" (UniqueName: \"kubernetes.io/projected/11c5b443-e562-448f-a540-2273f5ae86ca-kube-api-access-cfcmm\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.660238 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.660164 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.660407 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.660273 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.660407 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.660326 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.660407 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.660380 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.660554 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.660431 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c5b443-e562-448f-a540-2273f5ae86ca-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.761088 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.761046 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.761088 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.761089 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c5b443-e562-448f-a540-2273f5ae86ca-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.761353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.761117 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c5b443-e562-448f-a540-2273f5ae86ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.761353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.761138 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c5b443-e562-448f-a540-2273f5ae86ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.761353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.761166 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.761353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.761195 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfcmm\" (UniqueName: \"kubernetes.io/projected/11c5b443-e562-448f-a540-2273f5ae86ca-kube-api-access-cfcmm\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.761353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.761220 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.761353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.761300 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.761353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.761333 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.761745 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.761534 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.761745 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.761726 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.761843 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.761759 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.761913 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.761858 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.761970 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.761957 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c5b443-e562-448f-a540-2273f5ae86ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.763577 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.763554 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.763801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.763783 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c5b443-e562-448f-a540-2273f5ae86ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.770827 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.770805 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c5b443-e562-448f-a540-2273f5ae86ca-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.771984 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.771962 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfcmm\" (UniqueName: \"kubernetes.io/projected/11c5b443-e562-448f-a540-2273f5ae86ca-kube-api-access-cfcmm\") pod \"maas-default-gateway-openshift-default-769df7846c-9wbqx\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:11.944473 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:11.944436 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:12.223381 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:12.223348 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx"] Oct 31 03:25:12.225472 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:25:12.225439 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c5b443_e562_448f_a540_2273f5ae86ca.slice/crio-caeb1d1b14aa361ac55a7cdb2ff4eabf8ab735b64b3a6b9deef27a2498a581e2 WatchSource:0}: Error finding container caeb1d1b14aa361ac55a7cdb2ff4eabf8ab735b64b3a6b9deef27a2498a581e2: Status 404 returned error can't find the container with id caeb1d1b14aa361ac55a7cdb2ff4eabf8ab735b64b3a6b9deef27a2498a581e2 Oct 31 03:25:12.227923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:12.227889 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:25:12.228007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:12.227963 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:25:12.228007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:12.227990 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:25:12.330765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:12.330719 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" event={"ID":"11c5b443-e562-448f-a540-2273f5ae86ca","Type":"ContainerStarted","Data":"d37131b9f898f21b1425d43addcf6b69f1590f3344ea08bb6a598180a05fdf32"} Oct 31 03:25:12.330943 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:12.330835 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" event={"ID":"11c5b443-e562-448f-a540-2273f5ae86ca","Type":"ContainerStarted","Data":"caeb1d1b14aa361ac55a7cdb2ff4eabf8ab735b64b3a6b9deef27a2498a581e2"} Oct 31 03:25:12.350943 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:12.350867 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" podStartSLOduration=1.350847891 podStartE2EDuration="1.350847891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:12.349937229 +0000 UTC m=+546.884632183" watchObservedRunningTime="2026-10-31 03:25:12.350847891 +0000 UTC m=+546.885542846" Oct 31 03:25:12.947700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:12.947226 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:13.950843 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:13.950794 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-9wbqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 03:25:13.951332 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:13.950883 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" podUID="11c5b443-e562-448f-a540-2273f5ae86ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 03:25:14.948715 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:14.947956 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-9wbqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 03:25:14.948715 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:14.948023 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" podUID="11c5b443-e562-448f-a540-2273f5ae86ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 03:25:15.351792 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:15.351657 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:15.947250 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:15.947155 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-9wbqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 03:25:15.947664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:15.947625 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" podUID="11c5b443-e562-448f-a540-2273f5ae86ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 03:25:16.948701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:16.948105 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-9wbqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 03:25:16.948701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:16.948209 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" podUID="11c5b443-e562-448f-a540-2273f5ae86ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 03:25:17.136597 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:25:17.136464 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:25:17.719141 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:17.719100 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj"] Oct 31 03:25:17.719539 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:17.719488 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj" podUID="fe17388f-bfb3-4ff1-841f-93d137ca9bc0" containerName="manager" containerID="cri-o://a8d9ddd0766a541617d38ced4d0e45b3e1557f51de7895768d6f9682f1e45cbf" gracePeriod=30 Oct 31 03:25:17.955439 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:17.955075 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-9wbqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 03:25:17.955439 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:17.955129 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" podUID="11c5b443-e562-448f-a540-2273f5ae86ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 03:25:18.011958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:18.011932 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj" Oct 31 03:25:18.157373 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:18.157334 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe17388f-bfb3-4ff1-841f-93d137ca9bc0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe17388f-bfb3-4ff1-841f-93d137ca9bc0-metrics-certs\") pod \"fe17388f-bfb3-4ff1-841f-93d137ca9bc0\" (UID: \"fe17388f-bfb3-4ff1-841f-93d137ca9bc0\") " Oct 31 03:25:18.157549 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:18.157389 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe17388f-bfb3-4ff1-841f-93d137ca9bc0-kube-api-access-2m9dq\" (UniqueName: \"kubernetes.io/projected/fe17388f-bfb3-4ff1-841f-93d137ca9bc0-kube-api-access-2m9dq\") pod \"fe17388f-bfb3-4ff1-841f-93d137ca9bc0\" (UID: \"fe17388f-bfb3-4ff1-841f-93d137ca9bc0\") " Oct 31 03:25:18.159475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:18.159439 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe17388f-bfb3-4ff1-841f-93d137ca9bc0-metrics-certs" pod "fe17388f-bfb3-4ff1-841f-93d137ca9bc0" (UID: "fe17388f-bfb3-4ff1-841f-93d137ca9bc0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:25:18.159626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:18.159502 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe17388f-bfb3-4ff1-841f-93d137ca9bc0-kube-api-access-2m9dq" pod "fe17388f-bfb3-4ff1-841f-93d137ca9bc0" (UID: "fe17388f-bfb3-4ff1-841f-93d137ca9bc0"). InnerVolumeSpecName "kube-api-access-2m9dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:25:18.258922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:18.258840 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m9dq\" (UniqueName: \"kubernetes.io/projected/fe17388f-bfb3-4ff1-841f-93d137ca9bc0-kube-api-access-2m9dq\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:18.258922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:18.258870 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe17388f-bfb3-4ff1-841f-93d137ca9bc0-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:25:18.382237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:18.382204 5490 generic.go:356] "Generic (PLEG): container finished" podID="fe17388f-bfb3-4ff1-841f-93d137ca9bc0" containerID="a8d9ddd0766a541617d38ced4d0e45b3e1557f51de7895768d6f9682f1e45cbf" exitCode=0 Oct 31 03:25:18.382392 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:18.382251 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj" Oct 31 03:25:18.382392 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:18.382308 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj" event={"ID":"fe17388f-bfb3-4ff1-841f-93d137ca9bc0","Type":"ContainerDied","Data":"a8d9ddd0766a541617d38ced4d0e45b3e1557f51de7895768d6f9682f1e45cbf"} Oct 31 03:25:18.382392 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:18.382385 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj" event={"ID":"fe17388f-bfb3-4ff1-841f-93d137ca9bc0","Type":"ContainerDied","Data":"26e3737eb34c64eb4e7bcd6f6a9d267fece89e5bf01e9a4f6c9b69658fa448e8"} Oct 31 03:25:18.382515 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:18.382418 5490 scope.go:122] "RemoveContainer" containerID="a8d9ddd0766a541617d38ced4d0e45b3e1557f51de7895768d6f9682f1e45cbf" Oct 31 03:25:18.392295 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:18.392273 5490 scope.go:122] "RemoveContainer" containerID="a8d9ddd0766a541617d38ced4d0e45b3e1557f51de7895768d6f9682f1e45cbf" Oct 31 03:25:18.392566 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:25:18.392548 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d9ddd0766a541617d38ced4d0e45b3e1557f51de7895768d6f9682f1e45cbf\": container with ID starting with a8d9ddd0766a541617d38ced4d0e45b3e1557f51de7895768d6f9682f1e45cbf not found: ID does not exist" containerID="a8d9ddd0766a541617d38ced4d0e45b3e1557f51de7895768d6f9682f1e45cbf" Oct 31 03:25:18.392633 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:18.392575 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d9ddd0766a541617d38ced4d0e45b3e1557f51de7895768d6f9682f1e45cbf"} err="failed to get container status \"a8d9ddd0766a541617d38ced4d0e45b3e1557f51de7895768d6f9682f1e45cbf\": rpc error: code = NotFound desc = could not find container \"a8d9ddd0766a541617d38ced4d0e45b3e1557f51de7895768d6f9682f1e45cbf\": container with ID starting with a8d9ddd0766a541617d38ced4d0e45b3e1557f51de7895768d6f9682f1e45cbf not found: ID does not exist" Oct 31 03:25:18.405270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:18.405242 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj"] Oct 31 03:25:18.411852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:18.411825 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-gpzfj"] Oct 31 03:25:18.945502 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:18.945461 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-9wbqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 03:25:18.945745 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:18.945527 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" podUID="11c5b443-e562-448f-a540-2273f5ae86ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 03:25:19.465083 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:19.465046 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:19.574038 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:19.574006 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe17388f-bfb3-4ff1-841f-93d137ca9bc0" path="/var/lib/kubelet/pods/fe17388f-bfb3-4ff1-841f-93d137ca9bc0/volumes" Oct 31 03:25:19.971951 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:19.971914 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:20.496627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:20.496598 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:20.497101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:20.497046 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:20.497910 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:20.497890 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:25:21.547079 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:21.547047 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:21.691071 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:21.691032 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:23.153346 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:23.153290 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:23.336583 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:23.336547 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:36.226691 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:36.226647 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:47.127885 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:25:47.127835 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:25:47.833420 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:47.833380 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:48.034010 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.033976 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:48.460287 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.460245 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:48.684052 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.684016 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt"] Oct 31 03:25:48.684560 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.684539 5490 cpu_manager.go:414] "containerMap: removing container" podUID="fe17388f-bfb3-4ff1-841f-93d137ca9bc0" containerName="manager" Oct 31 03:25:48.684663 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.684562 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe17388f-bfb3-4ff1-841f-93d137ca9bc0" containerName="manager" Oct 31 03:25:48.684663 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.684658 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe17388f-bfb3-4ff1-841f-93d137ca9bc0" containerName="manager" Oct 31 03:25:48.687914 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.687892 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.711345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.711272 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d86a77-openshift-default-dockercfg-2xd8c\"" Oct 31 03:25:48.718398 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.718364 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt"] Oct 31 03:25:48.763866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.763830 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef5f9523-ec4a-4372-a327-009c9fc40563-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.764054 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.763885 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.764054 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.763990 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmmfb\" (UniqueName: \"kubernetes.io/projected/ef5f9523-ec4a-4372-a327-009c9fc40563-kube-api-access-vmmfb\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.764173 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.764052 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.764173 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.764102 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.764173 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.764123 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.764173 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.764166 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.764382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.764294 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.764431 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.764392 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.776780 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.776737 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:48.865556 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.865511 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.865775 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.865611 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.865775 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.865722 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef5f9523-ec4a-4372-a327-009c9fc40563-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.865886 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.865799 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.865886 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.865857 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmmfb\" (UniqueName: \"kubernetes.io/projected/ef5f9523-ec4a-4372-a327-009c9fc40563-kube-api-access-vmmfb\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.865989 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.865908 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.865989 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.865964 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.866094 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.865989 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.866094 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.866014 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.866210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.866131 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.866210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.866187 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.866374 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.866350 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.866828 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.866804 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef5f9523-ec4a-4372-a327-009c9fc40563-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.867052 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.866953 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.869535 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.869500 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.869747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.869719 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.884822 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.884789 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t"] Oct 31 03:25:48.888469 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.888441 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:48.899038 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.899008 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmmfb\" (UniqueName: \"kubernetes.io/projected/ef5f9523-ec4a-4372-a327-009c9fc40563-kube-api-access-vmmfb\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.899391 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.899368 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:48.906157 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.906135 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-6c6050-openshift-default-dockercfg-wsmcd\"" Oct 31 03:25:48.922593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.922548 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t"] Oct 31 03:25:48.967156 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.967066 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:48.967156 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.967142 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:48.967383 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.967274 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:48.967383 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.967313 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:48.967383 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.967339 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:48.967488 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.967445 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwlpp\" (UniqueName: \"kubernetes.io/projected/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-kube-api-access-pwlpp\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:48.967488 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.967481 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:48.967563 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.967506 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:48.967563 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:48.967550 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:49.001343 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.001307 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:49.027629 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.027574 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:49.090630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.089448 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:49.090630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.089498 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:49.090630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.089529 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:49.090630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.089620 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:49.090630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.089694 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:49.090630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.089756 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:49.090630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.089779 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:49.090630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.089807 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:49.090630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.089866 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwlpp\" (UniqueName: \"kubernetes.io/projected/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-kube-api-access-pwlpp\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:49.092615 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.092584 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:49.092894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.092869 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:49.094144 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.093613 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:49.094144 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.093947 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:49.094144 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.094102 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:49.101927 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.101875 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:49.104186 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.104164 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:49.155880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.155850 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwlpp\" (UniqueName: \"kubernetes.io/projected/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-kube-api-access-pwlpp\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:49.181333 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.181301 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:49.234338 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.234248 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:49.340604 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.340573 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt"] Oct 31 03:25:49.341727 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:25:49.341695 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5f9523_ec4a_4372_a327_009c9fc40563.slice/crio-c0a44c734d1c7a3e72e60683e7fe876ee329adc5db003ea6fc0273ab45413bd6 WatchSource:0}: Error finding container c0a44c734d1c7a3e72e60683e7fe876ee329adc5db003ea6fc0273ab45413bd6: Status 404 returned error can't find the container with id c0a44c734d1c7a3e72e60683e7fe876ee329adc5db003ea6fc0273ab45413bd6 Oct 31 03:25:49.344093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.344061 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:25:49.344198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.344135 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:25:49.344198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.344184 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:25:49.366188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.366146 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:49.538785 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.538759 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t"] Oct 31 03:25:49.541701 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:25:49.541654 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5907e061_56b9_48e5_b9a8_9d33fe49e5a1.slice/crio-c65b9cfab3a5c79e9f79bf40fec560834ede39aacf77b2438c1fdc56a485f9b1 WatchSource:0}: Error finding container c65b9cfab3a5c79e9f79bf40fec560834ede39aacf77b2438c1fdc56a485f9b1: Status 404 returned error can't find the container with id c65b9cfab3a5c79e9f79bf40fec560834ede39aacf77b2438c1fdc56a485f9b1 Oct 31 03:25:49.544270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.544232 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:25:49.544357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.544324 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:25:49.544396 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.544363 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:25:49.552660 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.552626 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" event={"ID":"5907e061-56b9-48e5-b9a8-9d33fe49e5a1","Type":"ContainerStarted","Data":"c65b9cfab3a5c79e9f79bf40fec560834ede39aacf77b2438c1fdc56a485f9b1"} Oct 31 03:25:49.554288 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.554259 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" event={"ID":"ef5f9523-ec4a-4372-a327-009c9fc40563","Type":"ContainerStarted","Data":"6416be98e88ea871330e8fb38a3a650458ed0aebc40ff174c0a8920654a91bc5"} Oct 31 03:25:49.554403 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.554315 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" event={"ID":"ef5f9523-ec4a-4372-a327-009c9fc40563","Type":"ContainerStarted","Data":"c0a44c734d1c7a3e72e60683e7fe876ee329adc5db003ea6fc0273ab45413bd6"} Oct 31 03:25:49.596290 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.592471 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" podStartSLOduration=1.592435256 podStartE2EDuration="1.592435256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:49.591501331 +0000 UTC m=+584.126196287" watchObservedRunningTime="2026-10-31 03:25:49.592435256 +0000 UTC m=+584.127130212" Oct 31 03:25:49.625459 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.625429 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:49.868601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:49.868513 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:50.001790 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:50.001738 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:50.007080 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:50.007050 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:50.560749 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:50.560712 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" event={"ID":"5907e061-56b9-48e5-b9a8-9d33fe49e5a1","Type":"ContainerStarted","Data":"6ee4efa6d5ab3bb9c86518bbfda5899d81cfff452cbce8623d5b21135c5f28cf"} Oct 31 03:25:50.561224 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:50.561081 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:50.562123 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:50.562103 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:25:50.597255 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:50.597202 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" podStartSLOduration=2.597187683 podStartE2EDuration="2.597187683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:50.594399616 +0000 UTC m=+585.129094570" watchObservedRunningTime="2026-10-31 03:25:50.597187683 +0000 UTC m=+585.131882637" Oct 31 03:25:51.235327 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:51.235292 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:51.242099 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:51.242069 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:51.567438 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:51.567350 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:51.573601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:51.573575 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:25:52.199368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:52.199323 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:52.624014 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:52.623919 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:52.868322 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:52.868279 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:53.109645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:53.109609 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:53.407636 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:53.407600 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:53.617806 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:53.617763 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:53.987800 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:53.987765 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv"] Oct 31 03:25:53.988723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:53.988664 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:53.991919 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:53.991899 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" Oct 31 03:25:54.007570 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.007547 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-d86a77-dockercfg-xfr64\"" Oct 31 03:25:54.007861 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.007841 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-d86a77\"" Oct 31 03:25:54.022499 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.022475 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv"] Oct 31 03:25:54.127556 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.127520 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2"] Oct 31 03:25:54.131280 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.131261 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" Oct 31 03:25:54.146883 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.146854 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2"] Oct 31 03:25:54.148909 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.148878 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6x2r\" (UniqueName: \"kubernetes.io/projected/5d9ae28c-045f-456f-bbd7-8105b6e0077d-kube-api-access-t6x2r\") pod \"payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv\" (UID: \"5d9ae28c-045f-456f-bbd7-8105b6e0077d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" Oct 31 03:25:54.149054 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.148941 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9ae28c-045f-456f-bbd7-8105b6e0077d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv\" (UID: \"5d9ae28c-045f-456f-bbd7-8105b6e0077d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" Oct 31 03:25:54.250353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.250268 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6x2r\" (UniqueName: \"kubernetes.io/projected/5d9ae28c-045f-456f-bbd7-8105b6e0077d-kube-api-access-t6x2r\") pod \"payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv\" (UID: \"5d9ae28c-045f-456f-bbd7-8105b6e0077d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" Oct 31 03:25:54.250353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.250332 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb68ad8-3da3-4469-9207-cb473d1dad9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2\" (UID: \"ceb68ad8-3da3-4469-9207-cb473d1dad9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" Oct 31 03:25:54.250599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.250368 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9ae28c-045f-456f-bbd7-8105b6e0077d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv\" (UID: \"5d9ae28c-045f-456f-bbd7-8105b6e0077d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" Oct 31 03:25:54.250599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.250391 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg4xp\" (UniqueName: \"kubernetes.io/projected/ceb68ad8-3da3-4469-9207-cb473d1dad9e-kube-api-access-fg4xp\") pod \"payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2\" (UID: \"ceb68ad8-3da3-4469-9207-cb473d1dad9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" Oct 31 03:25:54.251541 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.251513 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9ae28c-045f-456f-bbd7-8105b6e0077d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv\" (UID: \"5d9ae28c-045f-456f-bbd7-8105b6e0077d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" Oct 31 03:25:54.282124 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.282056 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6x2r\" (UniqueName: \"kubernetes.io/projected/5d9ae28c-045f-456f-bbd7-8105b6e0077d-kube-api-access-t6x2r\") pod \"payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv\" (UID: \"5d9ae28c-045f-456f-bbd7-8105b6e0077d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" Oct 31 03:25:54.282465 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.282373 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:54.305335 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.305296 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" Oct 31 03:25:54.351297 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.351259 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb68ad8-3da3-4469-9207-cb473d1dad9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2\" (UID: \"ceb68ad8-3da3-4469-9207-cb473d1dad9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" Oct 31 03:25:54.351497 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.351316 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg4xp\" (UniqueName: \"kubernetes.io/projected/ceb68ad8-3da3-4469-9207-cb473d1dad9e-kube-api-access-fg4xp\") pod \"payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2\" (UID: \"ceb68ad8-3da3-4469-9207-cb473d1dad9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" Oct 31 03:25:54.352264 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.352241 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb68ad8-3da3-4469-9207-cb473d1dad9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2\" (UID: \"ceb68ad8-3da3-4469-9207-cb473d1dad9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" Oct 31 03:25:54.388193 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.387441 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg4xp\" (UniqueName: \"kubernetes.io/projected/ceb68ad8-3da3-4469-9207-cb473d1dad9e-kube-api-access-fg4xp\") pod \"payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2\" (UID: \"ceb68ad8-3da3-4469-9207-cb473d1dad9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" Oct 31 03:25:54.446339 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.446302 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" Oct 31 03:25:54.612319 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.612293 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv"] Oct 31 03:25:54.615239 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:25:54.615209 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9ae28c_045f_456f_bbd7_8105b6e0077d.slice/crio-0852fdfbf1e5f6cd30fcb20ab22db9d956ac7e32a3763ef4c2cf00476af6d4ae WatchSource:0}: Error finding container 0852fdfbf1e5f6cd30fcb20ab22db9d956ac7e32a3763ef4c2cf00476af6d4ae: Status 404 returned error can't find the container with id 0852fdfbf1e5f6cd30fcb20ab22db9d956ac7e32a3763ef4c2cf00476af6d4ae Oct 31 03:25:54.639455 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.639429 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:54.742885 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:54.742855 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2"] Oct 31 03:25:54.745797 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:25:54.745751 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb68ad8_3da3_4469_9207_cb473d1dad9e.slice/crio-dde47236d35d30542b18cc50e65a8a674ff740006ff0b861d0efb92f4d06621f WatchSource:0}: Error finding container dde47236d35d30542b18cc50e65a8a674ff740006ff0b861d0efb92f4d06621f: Status 404 returned error can't find the container with id dde47236d35d30542b18cc50e65a8a674ff740006ff0b861d0efb92f4d06621f Oct 31 03:25:55.591077 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.591041 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" event={"ID":"ceb68ad8-3da3-4469-9207-cb473d1dad9e","Type":"ContainerStarted","Data":"126faaaf3ca3b600d22cb5fb1f588765b8b090637cd52a9d97e31ae85fec311e"} Oct 31 03:25:55.591562 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.591094 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" event={"ID":"ceb68ad8-3da3-4469-9207-cb473d1dad9e","Type":"ContainerStarted","Data":"dde47236d35d30542b18cc50e65a8a674ff740006ff0b861d0efb92f4d06621f"} Oct 31 03:25:55.591562 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.591424 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" Oct 31 03:25:55.592893 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.592867 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" event={"ID":"5d9ae28c-045f-456f-bbd7-8105b6e0077d","Type":"ContainerStarted","Data":"ea9dab4883d0e36f5ffd760db61792fa0b387dc6d8e994e50f2e46744896c43b"} Oct 31 03:25:55.592981 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.592909 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" event={"ID":"5d9ae28c-045f-456f-bbd7-8105b6e0077d","Type":"ContainerStarted","Data":"0852fdfbf1e5f6cd30fcb20ab22db9d956ac7e32a3763ef4c2cf00476af6d4ae"} Oct 31 03:25:55.592981 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.592942 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" Oct 31 03:25:55.659136 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.659096 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:55.672897 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.672847 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" podStartSLOduration=1.672832479 podStartE2EDuration="1.672832479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:55.67088926 +0000 UTC m=+590.205584215" watchObservedRunningTime="2026-10-31 03:25:55.672832479 +0000 UTC m=+590.207527434" Oct 31 03:25:55.692183 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.692144 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs"] Oct 31 03:25:55.696123 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.696095 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" Oct 31 03:25:55.717779 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.717745 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs"] Oct 31 03:25:55.718091 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.718053 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" podStartSLOduration=2.718041972 podStartE2EDuration="2.718041972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:55.712087735 +0000 UTC m=+590.246782702" watchObservedRunningTime="2026-10-31 03:25:55.718041972 +0000 UTC m=+590.252736921" Oct 31 03:25:55.727602 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.727567 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-090ebd-dockercfg-cqqqg\"" Oct 31 03:25:55.728407 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.728390 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-090ebd\"" Oct 31 03:25:55.874994 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.869250 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f75494-3ba5-460d-a96d-9ab2f4ffa395-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs\" (UID: \"e9f75494-3ba5-460d-a96d-9ab2f4ffa395\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" Oct 31 03:25:55.874994 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.869458 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qjhp\" (UniqueName: \"kubernetes.io/projected/e9f75494-3ba5-460d-a96d-9ab2f4ffa395-kube-api-access-9qjhp\") pod \"payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs\" (UID: \"e9f75494-3ba5-460d-a96d-9ab2f4ffa395\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" Oct 31 03:25:55.903671 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.903631 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:55.970480 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.970438 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qjhp\" (UniqueName: \"kubernetes.io/projected/e9f75494-3ba5-460d-a96d-9ab2f4ffa395-kube-api-access-9qjhp\") pod \"payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs\" (UID: \"e9f75494-3ba5-460d-a96d-9ab2f4ffa395\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" Oct 31 03:25:55.970671 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.970503 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f75494-3ba5-460d-a96d-9ab2f4ffa395-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs\" (UID: \"e9f75494-3ba5-460d-a96d-9ab2f4ffa395\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" Oct 31 03:25:55.971257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.971236 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f75494-3ba5-460d-a96d-9ab2f4ffa395-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs\" (UID: \"e9f75494-3ba5-460d-a96d-9ab2f4ffa395\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" Oct 31 03:25:55.982919 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:55.982896 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qjhp\" (UniqueName: \"kubernetes.io/projected/e9f75494-3ba5-460d-a96d-9ab2f4ffa395-kube-api-access-9qjhp\") pod \"payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs\" (UID: \"e9f75494-3ba5-460d-a96d-9ab2f4ffa395\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" Oct 31 03:25:56.011119 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.011079 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" Oct 31 03:25:56.292616 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.292578 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr"] Oct 31 03:25:56.297523 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.297500 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" Oct 31 03:25:56.323392 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.323230 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-d86a77-serving-cert\"" Oct 31 03:25:56.326720 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.326477 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs"] Oct 31 03:25:56.328817 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.328790 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr"] Oct 31 03:25:56.330269 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:25:56.330242 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f75494_3ba5_460d_a96d_9ab2f4ffa395.slice/crio-103e2faa078c1eec65101a799b0fff026756c504556ec3d323265645c2dfaf18 WatchSource:0}: Error finding container 103e2faa078c1eec65101a799b0fff026756c504556ec3d323265645c2dfaf18: Status 404 returned error can't find the container with id 103e2faa078c1eec65101a799b0fff026756c504556ec3d323265645c2dfaf18 Oct 31 03:25:56.476058 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.476024 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brcm7\" (UniqueName: \"kubernetes.io/projected/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-kube-api-access-brcm7\") pod \"maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr\" (UID: \"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" Oct 31 03:25:56.476226 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.476076 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr\" (UID: \"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" Oct 31 03:25:56.476268 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.476223 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr\" (UID: \"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" Oct 31 03:25:56.476303 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.476268 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-metrics-certs\") pod \"maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr\" (UID: \"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" Oct 31 03:25:56.479820 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.479788 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:56.577648 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.577558 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brcm7\" (UniqueName: \"kubernetes.io/projected/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-kube-api-access-brcm7\") pod \"maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr\" (UID: \"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" Oct 31 03:25:56.577648 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.577607 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr\" (UID: \"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" Oct 31 03:25:56.577901 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.577656 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr\" (UID: \"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" Oct 31 03:25:56.577901 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.577713 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-metrics-certs\") pod \"maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr\" (UID: \"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" Oct 31 03:25:56.578413 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.578390 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr\" (UID: \"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" Oct 31 03:25:56.580246 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.580222 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-metrics-certs\") pod \"maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr\" (UID: \"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" Oct 31 03:25:56.580332 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.580248 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr\" (UID: \"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" Oct 31 03:25:56.589604 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.589579 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brcm7\" (UniqueName: \"kubernetes.io/projected/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-kube-api-access-brcm7\") pod \"maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr\" (UID: \"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" Oct 31 03:25:56.600048 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.600012 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" event={"ID":"e9f75494-3ba5-460d-a96d-9ab2f4ffa395","Type":"ContainerStarted","Data":"37213b741452d2ad7a75e8ca947284454ac8bf5a856607630ef1d9adb48c0e8f"} Oct 31 03:25:56.600449 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.600076 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" event={"ID":"e9f75494-3ba5-460d-a96d-9ab2f4ffa395","Type":"ContainerStarted","Data":"103e2faa078c1eec65101a799b0fff026756c504556ec3d323265645c2dfaf18"} Oct 31 03:25:56.600449 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.600105 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" Oct 31 03:25:56.611645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.611612 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" Oct 31 03:25:56.643198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.643133 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" podStartSLOduration=1.643117235 podStartE2EDuration="1.643117235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:56.640324902 +0000 UTC m=+591.175019858" watchObservedRunningTime="2026-10-31 03:25:56.643117235 +0000 UTC m=+591.177812190" Oct 31 03:25:56.831188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.830975 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:56.917564 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.917538 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr"] Oct 31 03:25:56.919612 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:25:56.919576 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669f0bd3_7a5b_4e49_97d1_ca6fe5c241d0.slice/crio-904e52b38039b2773f2cedc6bf11ba6e2079373fdd88d2172719f1716f30e37f WatchSource:0}: Error finding container 904e52b38039b2773f2cedc6bf11ba6e2079373fdd88d2172719f1716f30e37f: Status 404 returned error can't find the container with id 904e52b38039b2773f2cedc6bf11ba6e2079373fdd88d2172719f1716f30e37f Oct 31 03:25:56.921113 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:56.921093 5490 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:25:57.154346 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.154316 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:57.167840 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.167809 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf"] Oct 31 03:25:57.172741 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.172722 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" Oct 31 03:25:57.197770 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.197743 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf"] Oct 31 03:25:57.206321 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.205785 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-f20dc4-dockercfg-wdx5k\"" Oct 31 03:25:57.208929 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.208910 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-f20dc4\"" Oct 31 03:25:57.286453 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.286418 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf50264-0d76-46ca-b2cb-998ae3bbeb1b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf\" (UID: \"4cf50264-0d76-46ca-b2cb-998ae3bbeb1b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" Oct 31 03:25:57.286637 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.286592 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ftq2\" (UniqueName: \"kubernetes.io/projected/4cf50264-0d76-46ca-b2cb-998ae3bbeb1b-kube-api-access-7ftq2\") pod \"payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf\" (UID: \"4cf50264-0d76-46ca-b2cb-998ae3bbeb1b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" Oct 31 03:25:57.388231 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.388181 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ftq2\" (UniqueName: \"kubernetes.io/projected/4cf50264-0d76-46ca-b2cb-998ae3bbeb1b-kube-api-access-7ftq2\") pod \"payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf\" (UID: \"4cf50264-0d76-46ca-b2cb-998ae3bbeb1b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" Oct 31 03:25:57.388430 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.388325 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf50264-0d76-46ca-b2cb-998ae3bbeb1b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf\" (UID: \"4cf50264-0d76-46ca-b2cb-998ae3bbeb1b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" Oct 31 03:25:57.389070 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.389042 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf50264-0d76-46ca-b2cb-998ae3bbeb1b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf\" (UID: \"4cf50264-0d76-46ca-b2cb-998ae3bbeb1b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" Oct 31 03:25:57.404591 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.404517 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ftq2\" (UniqueName: \"kubernetes.io/projected/4cf50264-0d76-46ca-b2cb-998ae3bbeb1b-kube-api-access-7ftq2\") pod \"payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf\" (UID: \"4cf50264-0d76-46ca-b2cb-998ae3bbeb1b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" Oct 31 03:25:57.450932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.450877 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:57.487538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.487495 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" Oct 31 03:25:57.606742 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.606707 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" event={"ID":"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0","Type":"ContainerStarted","Data":"47b429fcdff7c912f021dc68ed903c6f35dacd6b21c968a65d612fd15946b68f"} Oct 31 03:25:57.607117 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.606760 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" event={"ID":"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0","Type":"ContainerStarted","Data":"904e52b38039b2773f2cedc6bf11ba6e2079373fdd88d2172719f1716f30e37f"} Oct 31 03:25:57.607117 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.606885 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" Oct 31 03:25:57.644061 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.644010 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" podStartSLOduration=1.2693302499999999 podStartE2EDuration="1.643995368s" totalImagesPullingTime="374.665118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:57.643786492 +0000 UTC m=+592.178481446" watchObservedRunningTime="2026-10-31 03:25:57.643995368 +0000 UTC m=+592.178690323" Oct 31 03:25:57.785374 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.785281 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf"] Oct 31 03:25:57.796844 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:25:57.796810 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf50264_0d76_46ca_b2cb_998ae3bbeb1b.slice/crio-1f7d68da13054c6673e5381780b144fe7d75d2b8afd3e047df14d44337dbfb16 WatchSource:0}: Error finding container 1f7d68da13054c6673e5381780b144fe7d75d2b8afd3e047df14d44337dbfb16: Status 404 returned error can't find the container with id 1f7d68da13054c6673e5381780b144fe7d75d2b8afd3e047df14d44337dbfb16 Oct 31 03:25:57.905972 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.905935 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6"] Oct 31 03:25:57.909706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.909657 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" Oct 31 03:25:57.929091 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.928074 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6"] Oct 31 03:25:57.949760 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:57.949730 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-dda5cb-serving-cert\"" Oct 31 03:25:58.009375 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.009253 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:58.096164 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.096065 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a15588-0492-4ccc-96cc-ab09e1e1402d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6\" (UID: \"d2a15588-0492-4ccc-96cc-ab09e1e1402d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" Oct 31 03:25:58.096164 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.096111 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flqx7\" (UniqueName: \"kubernetes.io/projected/d2a15588-0492-4ccc-96cc-ab09e1e1402d-kube-api-access-flqx7\") pod \"maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6\" (UID: \"d2a15588-0492-4ccc-96cc-ab09e1e1402d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" Oct 31 03:25:58.096395 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.096290 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a15588-0492-4ccc-96cc-ab09e1e1402d-metrics-certs\") pod \"maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6\" (UID: \"d2a15588-0492-4ccc-96cc-ab09e1e1402d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" Oct 31 03:25:58.096459 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.096407 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a15588-0492-4ccc-96cc-ab09e1e1402d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6\" (UID: \"d2a15588-0492-4ccc-96cc-ab09e1e1402d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" Oct 31 03:25:58.197421 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.197378 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a15588-0492-4ccc-96cc-ab09e1e1402d-metrics-certs\") pod \"maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6\" (UID: \"d2a15588-0492-4ccc-96cc-ab09e1e1402d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" Oct 31 03:25:58.197598 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.197487 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a15588-0492-4ccc-96cc-ab09e1e1402d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6\" (UID: \"d2a15588-0492-4ccc-96cc-ab09e1e1402d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" Oct 31 03:25:58.197598 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.197538 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a15588-0492-4ccc-96cc-ab09e1e1402d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6\" (UID: \"d2a15588-0492-4ccc-96cc-ab09e1e1402d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" Oct 31 03:25:58.197598 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.197560 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flqx7\" (UniqueName: \"kubernetes.io/projected/d2a15588-0492-4ccc-96cc-ab09e1e1402d-kube-api-access-flqx7\") pod \"maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6\" (UID: \"d2a15588-0492-4ccc-96cc-ab09e1e1402d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" Oct 31 03:25:58.198208 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.198178 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a15588-0492-4ccc-96cc-ab09e1e1402d-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6\" (UID: \"d2a15588-0492-4ccc-96cc-ab09e1e1402d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" Oct 31 03:25:58.199923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.199901 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a15588-0492-4ccc-96cc-ab09e1e1402d-metrics-certs\") pod \"maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6\" (UID: \"d2a15588-0492-4ccc-96cc-ab09e1e1402d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" Oct 31 03:25:58.200029 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.199927 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a15588-0492-4ccc-96cc-ab09e1e1402d-maas-api-tls\") pod \"maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6\" (UID: \"d2a15588-0492-4ccc-96cc-ab09e1e1402d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" Oct 31 03:25:58.219100 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.219065 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flqx7\" (UniqueName: \"kubernetes.io/projected/d2a15588-0492-4ccc-96cc-ab09e1e1402d-kube-api-access-flqx7\") pod \"maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6\" (UID: \"d2a15588-0492-4ccc-96cc-ab09e1e1402d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" Oct 31 03:25:58.223864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.223841 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" Oct 31 03:25:58.295050 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.295012 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:58.529233 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.529206 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6"] Oct 31 03:25:58.531555 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:25:58.531509 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a15588_0492_4ccc_96cc_ab09e1e1402d.slice/crio-54692745dadfaf314648e822ff4f5685b80d7fd0dac5e443afc39fc52fcd66bb WatchSource:0}: Error finding container 54692745dadfaf314648e822ff4f5685b80d7fd0dac5e443afc39fc52fcd66bb: Status 404 returned error can't find the container with id 54692745dadfaf314648e822ff4f5685b80d7fd0dac5e443afc39fc52fcd66bb Oct 31 03:25:58.614208 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.614173 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" event={"ID":"d2a15588-0492-4ccc-96cc-ab09e1e1402d","Type":"ContainerStarted","Data":"54692745dadfaf314648e822ff4f5685b80d7fd0dac5e443afc39fc52fcd66bb"} Oct 31 03:25:58.615787 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.615760 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" event={"ID":"4cf50264-0d76-46ca-b2cb-998ae3bbeb1b","Type":"ContainerStarted","Data":"9a8d03166a4c32dde0efd5baaefb8adde72ac0c51ba156399e116bc5f90a3b3b"} Oct 31 03:25:58.615894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.615809 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" event={"ID":"4cf50264-0d76-46ca-b2cb-998ae3bbeb1b","Type":"ContainerStarted","Data":"1f7d68da13054c6673e5381780b144fe7d75d2b8afd3e047df14d44337dbfb16"} Oct 31 03:25:58.615894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.615859 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" Oct 31 03:25:58.653171 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.653108 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" podStartSLOduration=1.653090156 podStartE2EDuration="1.653090156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:58.651670965 +0000 UTC m=+593.186365922" watchObservedRunningTime="2026-10-31 03:25:58.653090156 +0000 UTC m=+593.187785112" Oct 31 03:25:58.655988 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.655953 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:58.998562 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:58.998525 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:59.226980 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.226890 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:59.335882 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.335851 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd"] Oct 31 03:25:59.340377 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.340351 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" Oct 31 03:25:59.352815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.352781 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-090ebd-serving-cert\"" Oct 31 03:25:59.356965 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.356939 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd"] Oct 31 03:25:59.412828 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.412792 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/288a58cd-9286-422b-bff8-15da47bc1a24-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd\" (UID: \"288a58cd-9286-422b-bff8-15da47bc1a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" Oct 31 03:25:59.413047 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.412845 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/288a58cd-9286-422b-bff8-15da47bc1a24-maas-api-tls\") pod \"maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd\" (UID: \"288a58cd-9286-422b-bff8-15da47bc1a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" Oct 31 03:25:59.413047 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.412871 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zxkf\" (UniqueName: \"kubernetes.io/projected/288a58cd-9286-422b-bff8-15da47bc1a24-kube-api-access-9zxkf\") pod \"maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd\" (UID: \"288a58cd-9286-422b-bff8-15da47bc1a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" Oct 31 03:25:59.413047 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.413004 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288a58cd-9286-422b-bff8-15da47bc1a24-metrics-certs\") pod \"maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd\" (UID: \"288a58cd-9286-422b-bff8-15da47bc1a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" Oct 31 03:25:59.478280 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.478192 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:59.514790 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.514653 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288a58cd-9286-422b-bff8-15da47bc1a24-metrics-certs\") pod \"maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd\" (UID: \"288a58cd-9286-422b-bff8-15da47bc1a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" Oct 31 03:25:59.514993 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.514897 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/288a58cd-9286-422b-bff8-15da47bc1a24-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd\" (UID: \"288a58cd-9286-422b-bff8-15da47bc1a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" Oct 31 03:25:59.514993 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.514937 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/288a58cd-9286-422b-bff8-15da47bc1a24-maas-api-tls\") pod \"maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd\" (UID: \"288a58cd-9286-422b-bff8-15da47bc1a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" Oct 31 03:25:59.514993 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.514964 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zxkf\" (UniqueName: \"kubernetes.io/projected/288a58cd-9286-422b-bff8-15da47bc1a24-kube-api-access-9zxkf\") pod \"maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd\" (UID: \"288a58cd-9286-422b-bff8-15da47bc1a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" Oct 31 03:25:59.515842 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.515815 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/288a58cd-9286-422b-bff8-15da47bc1a24-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd\" (UID: \"288a58cd-9286-422b-bff8-15da47bc1a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" Oct 31 03:25:59.518228 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.518207 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/288a58cd-9286-422b-bff8-15da47bc1a24-maas-api-tls\") pod \"maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd\" (UID: \"288a58cd-9286-422b-bff8-15da47bc1a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" Oct 31 03:25:59.518614 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.518598 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288a58cd-9286-422b-bff8-15da47bc1a24-metrics-certs\") pod \"maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd\" (UID: \"288a58cd-9286-422b-bff8-15da47bc1a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" Oct 31 03:25:59.528700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.528659 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zxkf\" (UniqueName: \"kubernetes.io/projected/288a58cd-9286-422b-bff8-15da47bc1a24-kube-api-access-9zxkf\") pod \"maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd\" (UID: \"288a58cd-9286-422b-bff8-15da47bc1a24\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" Oct 31 03:25:59.622629 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.622596 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" event={"ID":"d2a15588-0492-4ccc-96cc-ab09e1e1402d","Type":"ContainerStarted","Data":"a197cd4f451c0d78160082e2f7f2926831f3c17d475609a4a452eeeb0ac098a1"} Oct 31 03:25:59.623140 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.622840 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" Oct 31 03:25:59.653845 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.653815 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" Oct 31 03:25:59.655310 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.655268 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" podStartSLOduration=2.279754045 podStartE2EDuration="2.65525636s" totalImagesPullingTime="375.502315ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:59.652051318 +0000 UTC m=+594.186746276" watchObservedRunningTime="2026-10-31 03:25:59.65525636 +0000 UTC m=+594.189951378" Oct 31 03:25:59.760403 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.760302 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:25:59.960415 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:25:59.960389 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd"] Oct 31 03:25:59.962301 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:25:59.962271 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288a58cd_9286_422b_bff8_15da47bc1a24.slice/crio-beb262ecbd696cc201614255b98292ef089c7fab793319d21c7901613b2dd7c4 WatchSource:0}: Error finding container beb262ecbd696cc201614255b98292ef089c7fab793319d21c7901613b2dd7c4: Status 404 returned error can't find the container with id beb262ecbd696cc201614255b98292ef089c7fab793319d21c7901613b2dd7c4 Oct 31 03:26:00.410295 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:00.410245 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:00.621849 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:00.619488 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:00.634124 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:00.634078 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" event={"ID":"288a58cd-9286-422b-bff8-15da47bc1a24","Type":"ContainerStarted","Data":"57363db53745908cd4ba258785ed632f5eaabe1f44fa4682c42ca106a42f93b4"} Oct 31 03:26:00.634542 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:00.634143 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" event={"ID":"288a58cd-9286-422b-bff8-15da47bc1a24","Type":"ContainerStarted","Data":"beb262ecbd696cc201614255b98292ef089c7fab793319d21c7901613b2dd7c4"} Oct 31 03:26:00.634542 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:00.634354 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" Oct 31 03:26:00.672238 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:00.672135 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" podStartSLOduration=1.198622501 podStartE2EDuration="1.672120053s" totalImagesPullingTime="473.497552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:00.671917572 +0000 UTC m=+595.206612539" watchObservedRunningTime="2026-10-31 03:26:00.672120053 +0000 UTC m=+595.206815008" Oct 31 03:26:00.813878 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:00.813831 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:01.045178 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:01.045099 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:01.206443 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:01.206182 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:01.495340 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:01.495301 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:01.635949 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:01.635910 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:01.880164 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:01.880069 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:02.408342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:02.408307 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5"] Oct 31 03:26:02.412511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:02.412478 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" Oct 31 03:26:02.415894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:02.415870 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5"] Oct 31 03:26:02.416527 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:02.416504 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-bcc7a4-serving-cert\"" Oct 31 03:26:02.456697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:02.456649 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531c8516-8a62-4add-a9bb-15b02ebb6f5d-metrics-certs\") pod \"maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5\" (UID: \"531c8516-8a62-4add-a9bb-15b02ebb6f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" Oct 31 03:26:02.456874 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:02.456730 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zfzn\" (UniqueName: \"kubernetes.io/projected/531c8516-8a62-4add-a9bb-15b02ebb6f5d-kube-api-access-7zfzn\") pod \"maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5\" (UID: \"531c8516-8a62-4add-a9bb-15b02ebb6f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" Oct 31 03:26:02.456874 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:02.456845 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531c8516-8a62-4add-a9bb-15b02ebb6f5d-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5\" (UID: \"531c8516-8a62-4add-a9bb-15b02ebb6f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" Oct 31 03:26:02.456957 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:02.456904 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531c8516-8a62-4add-a9bb-15b02ebb6f5d-maas-api-tls\") pod \"maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5\" (UID: \"531c8516-8a62-4add-a9bb-15b02ebb6f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" Oct 31 03:26:02.488964 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:02.488926 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:02.557878 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:02.557839 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zfzn\" (UniqueName: \"kubernetes.io/projected/531c8516-8a62-4add-a9bb-15b02ebb6f5d-kube-api-access-7zfzn\") pod \"maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5\" (UID: \"531c8516-8a62-4add-a9bb-15b02ebb6f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" Oct 31 03:26:02.558076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:02.557898 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531c8516-8a62-4add-a9bb-15b02ebb6f5d-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5\" (UID: \"531c8516-8a62-4add-a9bb-15b02ebb6f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" Oct 31 03:26:02.558076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:02.557940 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531c8516-8a62-4add-a9bb-15b02ebb6f5d-maas-api-tls\") pod \"maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5\" (UID: \"531c8516-8a62-4add-a9bb-15b02ebb6f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" Oct 31 03:26:02.558194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:02.558116 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531c8516-8a62-4add-a9bb-15b02ebb6f5d-metrics-certs\") pod \"maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5\" (UID: \"531c8516-8a62-4add-a9bb-15b02ebb6f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" Oct 31 03:26:02.558642 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:02.558611 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531c8516-8a62-4add-a9bb-15b02ebb6f5d-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5\" (UID: \"531c8516-8a62-4add-a9bb-15b02ebb6f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" Oct 31 03:26:02.560482 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:02.560464 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531c8516-8a62-4add-a9bb-15b02ebb6f5d-maas-api-tls\") pod \"maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5\" (UID: \"531c8516-8a62-4add-a9bb-15b02ebb6f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" Oct 31 03:26:02.561318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:02.561299 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531c8516-8a62-4add-a9bb-15b02ebb6f5d-metrics-certs\") pod \"maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5\" (UID: \"531c8516-8a62-4add-a9bb-15b02ebb6f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" Oct 31 03:26:02.566479 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:02.566450 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zfzn\" (UniqueName: \"kubernetes.io/projected/531c8516-8a62-4add-a9bb-15b02ebb6f5d-kube-api-access-7zfzn\") pod \"maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5\" (UID: \"531c8516-8a62-4add-a9bb-15b02ebb6f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" Oct 31 03:26:02.730564 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:02.730476 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" Oct 31 03:26:02.775484 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:02.775451 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:03.010122 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.010043 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5"] Oct 31 03:26:03.015387 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:26:03.015363 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531c8516_8a62_4add_a9bb_15b02ebb6f5d.slice/crio-ffa10155a62575e746871081b0d464a9737378c1e97c64140ebe33a889587b96 WatchSource:0}: Error finding container ffa10155a62575e746871081b0d464a9737378c1e97c64140ebe33a889587b96: Status 404 returned error can't find the container with id ffa10155a62575e746871081b0d464a9737378c1e97c64140ebe33a889587b96 Oct 31 03:26:03.302836 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.302746 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n"] Oct 31 03:26:03.308654 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.308630 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" Oct 31 03:26:03.318753 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.318585 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n"] Oct 31 03:26:03.320497 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.320474 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-7f3159-serving-cert\"" Oct 31 03:26:03.369323 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.369284 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1303047d-c782-41c7-bc35-d3298c9ef224-metrics-certs\") pod \"maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n\" (UID: \"1303047d-c782-41c7-bc35-d3298c9ef224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" Oct 31 03:26:03.369525 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.369501 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmwxm\" (UniqueName: \"kubernetes.io/projected/1303047d-c782-41c7-bc35-d3298c9ef224-kube-api-access-hmwxm\") pod \"maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n\" (UID: \"1303047d-c782-41c7-bc35-d3298c9ef224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" Oct 31 03:26:03.369592 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.369537 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1303047d-c782-41c7-bc35-d3298c9ef224-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n\" (UID: \"1303047d-c782-41c7-bc35-d3298c9ef224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" Oct 31 03:26:03.369638 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.369608 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1303047d-c782-41c7-bc35-d3298c9ef224-maas-api-tls\") pod \"maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n\" (UID: \"1303047d-c782-41c7-bc35-d3298c9ef224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" Oct 31 03:26:03.411649 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.411615 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:03.471060 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.471027 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1303047d-c782-41c7-bc35-d3298c9ef224-maas-api-tls\") pod \"maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n\" (UID: \"1303047d-c782-41c7-bc35-d3298c9ef224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" Oct 31 03:26:03.471274 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.471108 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1303047d-c782-41c7-bc35-d3298c9ef224-metrics-certs\") pod \"maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n\" (UID: \"1303047d-c782-41c7-bc35-d3298c9ef224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" Oct 31 03:26:03.471367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.471275 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmwxm\" (UniqueName: \"kubernetes.io/projected/1303047d-c782-41c7-bc35-d3298c9ef224-kube-api-access-hmwxm\") pod \"maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n\" (UID: \"1303047d-c782-41c7-bc35-d3298c9ef224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" Oct 31 03:26:03.471367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.471308 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1303047d-c782-41c7-bc35-d3298c9ef224-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n\" (UID: \"1303047d-c782-41c7-bc35-d3298c9ef224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" Oct 31 03:26:03.482761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.472068 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1303047d-c782-41c7-bc35-d3298c9ef224-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n\" (UID: \"1303047d-c782-41c7-bc35-d3298c9ef224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" Oct 31 03:26:03.482761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.477404 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1303047d-c782-41c7-bc35-d3298c9ef224-metrics-certs\") pod \"maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n\" (UID: \"1303047d-c782-41c7-bc35-d3298c9ef224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" Oct 31 03:26:03.482761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.477581 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1303047d-c782-41c7-bc35-d3298c9ef224-maas-api-tls\") pod \"maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n\" (UID: \"1303047d-c782-41c7-bc35-d3298c9ef224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" Oct 31 03:26:03.486241 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.486195 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmwxm\" (UniqueName: \"kubernetes.io/projected/1303047d-c782-41c7-bc35-d3298c9ef224-kube-api-access-hmwxm\") pod \"maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n\" (UID: \"1303047d-c782-41c7-bc35-d3298c9ef224\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" Oct 31 03:26:03.556361 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.556266 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:03.622258 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.622226 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" Oct 31 03:26:03.627488 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.627463 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" Oct 31 03:26:03.658328 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.658288 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" event={"ID":"531c8516-8a62-4add-a9bb-15b02ebb6f5d","Type":"ContainerStarted","Data":"db46fc3bf6d0814a7e7d227a3141110ba8ea1f0ac6d031c66ca2a2318bfa2ee1"} Oct 31 03:26:03.658476 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.658357 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" event={"ID":"531c8516-8a62-4add-a9bb-15b02ebb6f5d","Type":"ContainerStarted","Data":"ffa10155a62575e746871081b0d464a9737378c1e97c64140ebe33a889587b96"} Oct 31 03:26:03.659286 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.659264 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" Oct 31 03:26:03.684138 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.683762 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" podStartSLOduration=1.285553335 podStartE2EDuration="1.683742491s" totalImagesPullingTime="398.189156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:03.678869944 +0000 UTC m=+598.213564892" watchObservedRunningTime="2026-10-31 03:26:03.683742491 +0000 UTC m=+598.218437447" Oct 31 03:26:03.919411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:03.919384 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n"] Oct 31 03:26:03.921115 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:26:03.921090 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1303047d_c782_41c7_bc35_d3298c9ef224.slice/crio-dbf657a00c699bcc04cd3b856f64e09f374b097469824cd5d6a27d6662ed381d WatchSource:0}: Error finding container dbf657a00c699bcc04cd3b856f64e09f374b097469824cd5d6a27d6662ed381d: Status 404 returned error can't find the container with id dbf657a00c699bcc04cd3b856f64e09f374b097469824cd5d6a27d6662ed381d Oct 31 03:26:04.318029 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:04.317933 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:04.668070 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:04.668028 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" event={"ID":"1303047d-c782-41c7-bc35-d3298c9ef224","Type":"ContainerStarted","Data":"3d916d97247ba656eebfa4e5db031bdc9b1855ddaeb6bd8d2946f85ddf648907"} Oct 31 03:26:04.668249 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:04.668102 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" Oct 31 03:26:04.668249 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:04.668125 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" event={"ID":"1303047d-c782-41c7-bc35-d3298c9ef224","Type":"ContainerStarted","Data":"dbf657a00c699bcc04cd3b856f64e09f374b097469824cd5d6a27d6662ed381d"} Oct 31 03:26:04.695367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:04.695312 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" podStartSLOduration=1.183502945 podStartE2EDuration="1.695296662s" totalImagesPullingTime="511.793717ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:04.692295882 +0000 UTC m=+599.226990837" watchObservedRunningTime="2026-10-31 03:26:04.695296662 +0000 UTC m=+599.229991617" Oct 31 03:26:05.009345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:05.009243 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:05.167187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:05.167128 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:05.314144 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:05.314045 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:05.666657 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:05.666618 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" Oct 31 03:26:06.375307 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:06.375268 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:06.603395 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:06.603366 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" Oct 31 03:26:06.603590 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:06.603555 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" Oct 31 03:26:06.663787 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:06.663743 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" Oct 31 03:26:07.152321 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:07.152280 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:07.610110 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:07.610076 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" Oct 31 03:26:07.615650 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:07.615620 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:07.756182 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:07.756146 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:07.913529 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:07.913495 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:08.358145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:08.356923 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:08.894648 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:08.894613 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:09.046309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:09.046220 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:09.189957 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:09.189917 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:09.501099 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:09.500994 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:09.641911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:09.641875 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" Oct 31 03:26:09.992502 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:09.992468 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:10.694772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:10.694744 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" Oct 31 03:26:10.695542 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:10.695520 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" Oct 31 03:26:11.002061 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:11.001966 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:11.885423 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:11.885375 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:12.622041 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:12.621999 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:12.769029 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:12.768981 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:13.420465 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:13.420423 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:13.856067 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:13.855952 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:14.229193 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:14.229160 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:14.757667 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:14.757635 5490 generic.go:356] "Generic (PLEG): container finished" podID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerID="6416be98e88ea871330e8fb38a3a650458ed0aebc40ff174c0a8920654a91bc5" exitCode=0 Oct 31 03:26:14.757875 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:14.757712 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" event={"ID":"ef5f9523-ec4a-4372-a327-009c9fc40563","Type":"ContainerDied","Data":"6416be98e88ea871330e8fb38a3a650458ed0aebc40ff174c0a8920654a91bc5"} Oct 31 03:26:14.758143 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:14.758127 5490 scope.go:122] "RemoveContainer" containerID="6416be98e88ea871330e8fb38a3a650458ed0aebc40ff174c0a8920654a91bc5" Oct 31 03:26:14.759601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:14.759574 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:26:14.759701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:14.759642 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:26:14.759746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:14.759701 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:26:15.171840 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:15.171798 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:15.304296 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:15.304261 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:15.464770 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:15.462604 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:15.784088 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:15.783993 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" event={"ID":"ef5f9523-ec4a-4372-a327-009c9fc40563","Type":"ContainerStarted","Data":"8212151b6af29b681d3346cd5171621f62899608c3f910e235e612a058935331"} Oct 31 03:26:16.002069 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:16.002022 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:26:16.002788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:16.002747 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 31 03:26:16.002926 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:16.002883 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" podUID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 31 03:26:16.299727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:16.299690 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:17.002779 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:17.002732 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 31 03:26:17.002963 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:17.002804 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" podUID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 31 03:26:17.148455 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:26:17.148416 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:26:17.189176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:17.189148 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:17.960556 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:17.960521 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:18.002552 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:18.002518 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 31 03:26:18.002802 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:18.002778 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" podUID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 31 03:26:18.111020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:18.110983 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:18.256756 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:18.256169 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:18.578754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:18.577666 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:18.970382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:18.970339 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:19.002200 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:19.002153 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 31 03:26:19.002405 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:19.002235 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" podUID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 31 03:26:19.132554 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:19.132509 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:19.364538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:19.364447 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:20.002313 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:20.002165 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 31 03:26:20.002313 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:20.002224 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" podUID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 31 03:26:20.563701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:20.562793 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:26:20.668012 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:20.661062 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:20.800813 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:20.800770 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:21.002364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:21.002327 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 31 03:26:21.002787 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:21.002409 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" podUID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 31 03:26:21.294627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:21.294522 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:22.007191 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:22.007139 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:26:22.008757 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:22.008721 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:26:22.094254 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:22.094218 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:22.608870 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:22.608830 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:26:22.609052 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:22.608897 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" podUID="5907e061-56b9-48e5-b9a8-9d33fe49e5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:26:22.773205 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:22.773163 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:22.980685 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:22.980637 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:23.114300 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:23.114266 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:23.510395 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:23.510349 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:23.757488 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:23.757451 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:23.913547 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:23.913518 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:24.391729 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:24.391652 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:24.844489 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:24.844459 5490 generic.go:356] "Generic (PLEG): container finished" podID="5907e061-56b9-48e5-b9a8-9d33fe49e5a1" containerID="6ee4efa6d5ab3bb9c86518bbfda5899d81cfff452cbce8623d5b21135c5f28cf" exitCode=0 Oct 31 03:26:24.844661 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:24.844537 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" event={"ID":"5907e061-56b9-48e5-b9a8-9d33fe49e5a1","Type":"ContainerDied","Data":"6ee4efa6d5ab3bb9c86518bbfda5899d81cfff452cbce8623d5b21135c5f28cf"} Oct 31 03:26:24.845051 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:24.845029 5490 scope.go:122] "RemoveContainer" containerID="6ee4efa6d5ab3bb9c86518bbfda5899d81cfff452cbce8623d5b21135c5f28cf" Oct 31 03:26:24.846469 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:24.846444 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:26:24.846524 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:24.846507 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:26:24.846559 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:24.846542 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:26:25.236883 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:25.236845 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:25.858932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:25.858897 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" event={"ID":"5907e061-56b9-48e5-b9a8-9d33fe49e5a1","Type":"ContainerStarted","Data":"40d436983fdb913d5e45a8e8fbe4af9c11ee71a8c3b547a04e6f3bde6a64fb95"} Oct 31 03:26:26.033937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:26.032840 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:26.238248 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:26.238122 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:26:26.238576 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:26.238551 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 31 03:26:26.238762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:26.238736 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" podUID="5907e061-56b9-48e5-b9a8-9d33fe49e5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 31 03:26:27.174966 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:27.174927 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:27.255726 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:27.255263 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 31 03:26:27.255726 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:27.255323 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" podUID="5907e061-56b9-48e5-b9a8-9d33fe49e5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 31 03:26:27.899484 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:27.899445 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:28.239778 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:28.239738 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 31 03:26:28.240402 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:28.240364 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" podUID="5907e061-56b9-48e5-b9a8-9d33fe49e5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 31 03:26:28.648966 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:28.648691 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:29.242702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:29.241071 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 31 03:26:29.242702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:29.241140 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" podUID="5907e061-56b9-48e5-b9a8-9d33fe49e5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 31 03:26:29.386492 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:29.386451 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:30.095530 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:30.095482 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:30.239702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:30.237109 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 31 03:26:30.239702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:30.237194 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" podUID="5907e061-56b9-48e5-b9a8-9d33fe49e5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 31 03:26:30.776777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:30.776738 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:31.239023 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:31.238994 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:26:31.239325 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:31.239300 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:26:31.240159 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:31.240135 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:26:31.620187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:31.620080 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:32.318469 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:32.318432 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:33.086400 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:33.086362 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:33.788479 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:33.788441 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:33.943808 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:33.943773 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:34.727522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:34.727484 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:35.505430 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:35.505387 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:36.337321 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:36.337275 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:37.538174 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:37.538137 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:37.703751 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:37.703664 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:38.205510 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:38.205471 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:39.022972 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:39.022936 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:39.869748 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:39.869709 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:40.585518 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:40.585485 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:40.724848 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:40.724810 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:41.496498 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:41.496461 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:42.538157 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:42.538113 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:43.325366 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:43.325330 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:44.011897 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:44.011864 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:44.727244 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:44.727197 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:45.317309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.317275 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll"] Oct 31 03:26:45.351429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.351397 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:45.378140 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.377029 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll"] Oct 31 03:26:45.378140 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.377874 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-dda5cb\"/\"kube-root-ca.crt\"" Oct 31 03:26:45.378140 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.377985 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-dda5cb\"/\"default-dockercfg-2qpsx\"" Oct 31 03:26:45.378140 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.378034 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-dda5cb\"/\"openshift-service-ca.crt\"" Oct 31 03:26:45.378140 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.378101 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-dda5cb\"/\"premium-w4-dda5cb-kserve-self-signed-certs\"" Oct 31 03:26:45.466696 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.456960 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:45.488159 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.488119 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-model-cache\") pod \"premium-w4-dda5cb-kserve-796ff5cf99-g2qll\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:45.488349 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.488184 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e4007d-c749-49c9-b6b2-301e57a795b3-tls-certs\") pod \"premium-w4-dda5cb-kserve-796ff5cf99-g2qll\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:45.488349 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.488206 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-home\") pod \"premium-w4-dda5cb-kserve-796ff5cf99-g2qll\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:45.488349 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.488334 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcjsl\" (UniqueName: \"kubernetes.io/projected/51e4007d-c749-49c9-b6b2-301e57a795b3-kube-api-access-hcjsl\") pod \"premium-w4-dda5cb-kserve-796ff5cf99-g2qll\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:45.488528 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.488374 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-tmp-dir\") pod \"premium-w4-dda5cb-kserve-796ff5cf99-g2qll\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:45.488528 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.488431 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-dshm\") pod \"premium-w4-dda5cb-kserve-796ff5cf99-g2qll\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:45.589785 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.589697 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-model-cache\") pod \"premium-w4-dda5cb-kserve-796ff5cf99-g2qll\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:45.589970 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.589794 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e4007d-c749-49c9-b6b2-301e57a795b3-tls-certs\") pod \"premium-w4-dda5cb-kserve-796ff5cf99-g2qll\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:45.589970 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.589819 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-home\") pod \"premium-w4-dda5cb-kserve-796ff5cf99-g2qll\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:45.589970 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.589876 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcjsl\" (UniqueName: \"kubernetes.io/projected/51e4007d-c749-49c9-b6b2-301e57a795b3-kube-api-access-hcjsl\") pod \"premium-w4-dda5cb-kserve-796ff5cf99-g2qll\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:45.589970 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.589927 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-tmp-dir\") pod \"premium-w4-dda5cb-kserve-796ff5cf99-g2qll\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:45.589970 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.589961 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-dshm\") pod \"premium-w4-dda5cb-kserve-796ff5cf99-g2qll\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:45.590236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.590157 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-model-cache\") pod \"premium-w4-dda5cb-kserve-796ff5cf99-g2qll\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:45.590236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.590200 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-home\") pod \"premium-w4-dda5cb-kserve-796ff5cf99-g2qll\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:45.590356 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.590275 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-tmp-dir\") pod \"premium-w4-dda5cb-kserve-796ff5cf99-g2qll\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:45.592415 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.592387 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-dshm\") pod \"premium-w4-dda5cb-kserve-796ff5cf99-g2qll\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:45.592744 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.592723 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e4007d-c749-49c9-b6b2-301e57a795b3-tls-certs\") pod \"premium-w4-dda5cb-kserve-796ff5cf99-g2qll\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:45.598422 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.598400 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcjsl\" (UniqueName: \"kubernetes.io/projected/51e4007d-c749-49c9-b6b2-301e57a795b3-kube-api-access-hcjsl\") pod \"premium-w4-dda5cb-kserve-796ff5cf99-g2qll\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:45.680128 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.680088 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:45.748377 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:45.748338 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:46.032644 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:46.032523 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll"] Oct 31 03:26:46.034338 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:26:46.034305 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51e4007d_c749_49c9_b6b2_301e57a795b3.slice/crio-b7f6023e0738047ef7a93100ae24364e6245dee56d3a402f3fba1032ad45ac91 WatchSource:0}: Error finding container b7f6023e0738047ef7a93100ae24364e6245dee56d3a402f3fba1032ad45ac91: Status 404 returned error can't find the container with id b7f6023e0738047ef7a93100ae24364e6245dee56d3a402f3fba1032ad45ac91 Oct 31 03:26:46.415208 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:46.415176 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:46.991627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:46.991584 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" event={"ID":"51e4007d-c749-49c9-b6b2-301e57a795b3","Type":"ContainerStarted","Data":"3be94f0b6a094534c49f97c3d94fb16e081ec07f1e0e0c3155615f216f11ee30"} Oct 31 03:26:46.991831 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:46.991638 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" event={"ID":"51e4007d-c749-49c9-b6b2-301e57a795b3","Type":"ContainerStarted","Data":"b7f6023e0738047ef7a93100ae24364e6245dee56d3a402f3fba1032ad45ac91"} Oct 31 03:26:47.012915 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:47.012864 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podStartSLOduration=2.012851304 podStartE2EDuration="2.012851304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:47.010507916 +0000 UTC m=+641.545202871" watchObservedRunningTime="2026-10-31 03:26:47.012851304 +0000 UTC m=+641.547546291" Oct 31 03:26:47.128081 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:26:47.128040 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:26:47.209897 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:47.209865 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:47.749341 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:47.749301 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:47.901890 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:47.901845 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:48.498161 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:48.498125 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:48.642777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:48.642727 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:49.290089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:49.290053 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:50.035894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:50.035859 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:50.814317 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:50.814261 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:51.506794 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:51.506762 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:52.203861 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.203817 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:52.396509 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.396464 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl"] Oct 31 03:26:52.400736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.400711 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:52.409779 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.409754 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-bcc7a4\"/\"default-dockercfg-bvtsj\"" Oct 31 03:26:52.409779 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.409759 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-bcc7a4\"/\"premium-w5-bcc7a4-kserve-self-signed-certs\"" Oct 31 03:26:52.409977 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.409810 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-bcc7a4\"/\"kube-root-ca.crt\"" Oct 31 03:26:52.409977 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.409824 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-bcc7a4\"/\"openshift-service-ca.crt\"" Oct 31 03:26:52.424163 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.424131 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl"] Oct 31 03:26:52.467261 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.467183 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-tmp-dir\") pod \"premium-w5-bcc7a4-kserve-864876c669-wlghl\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:52.467261 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.467244 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n246z\" (UniqueName: \"kubernetes.io/projected/1712a46e-a034-40f6-a276-09a907dd082f-kube-api-access-n246z\") pod \"premium-w5-bcc7a4-kserve-864876c669-wlghl\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:52.467444 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.467328 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1712a46e-a034-40f6-a276-09a907dd082f-tls-certs\") pod \"premium-w5-bcc7a4-kserve-864876c669-wlghl\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:52.467444 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.467378 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-home\") pod \"premium-w5-bcc7a4-kserve-864876c669-wlghl\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:52.467444 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.467416 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-dshm\") pod \"premium-w5-bcc7a4-kserve-864876c669-wlghl\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:52.467550 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.467534 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-model-cache\") pod \"premium-w5-bcc7a4-kserve-864876c669-wlghl\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:52.566214 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.566177 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:52.568462 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.568433 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-dshm\") pod \"premium-w5-bcc7a4-kserve-864876c669-wlghl\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:52.568618 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.568489 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-model-cache\") pod \"premium-w5-bcc7a4-kserve-864876c669-wlghl\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:52.568618 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.568563 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-tmp-dir\") pod \"premium-w5-bcc7a4-kserve-864876c669-wlghl\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:52.568618 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.568600 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n246z\" (UniqueName: \"kubernetes.io/projected/1712a46e-a034-40f6-a276-09a907dd082f-kube-api-access-n246z\") pod \"premium-w5-bcc7a4-kserve-864876c669-wlghl\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:52.568817 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.568626 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1712a46e-a034-40f6-a276-09a907dd082f-tls-certs\") pod \"premium-w5-bcc7a4-kserve-864876c669-wlghl\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:52.568817 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.568666 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-home\") pod \"premium-w5-bcc7a4-kserve-864876c669-wlghl\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:52.569040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.569018 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-model-cache\") pod \"premium-w5-bcc7a4-kserve-864876c669-wlghl\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:52.569107 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.569055 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-tmp-dir\") pod \"premium-w5-bcc7a4-kserve-864876c669-wlghl\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:52.569107 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.569098 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-home\") pod \"premium-w5-bcc7a4-kserve-864876c669-wlghl\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:52.571583 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.571562 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1712a46e-a034-40f6-a276-09a907dd082f-tls-certs\") pod \"premium-w5-bcc7a4-kserve-864876c669-wlghl\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:52.573518 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.573492 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-dshm\") pod \"premium-w5-bcc7a4-kserve-864876c669-wlghl\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:52.594051 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.594018 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n246z\" (UniqueName: \"kubernetes.io/projected/1712a46e-a034-40f6-a276-09a907dd082f-kube-api-access-n246z\") pod \"premium-w5-bcc7a4-kserve-864876c669-wlghl\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:52.714702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:52.714461 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:53.023424 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:53.023321 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl"] Oct 31 03:26:53.028263 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:26:53.028210 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1712a46e_a034_40f6_a276_09a907dd082f.slice/crio-e3c59306b82ea5223a1848b314404162d3337758f1f1eb395202afa94d2f337a WatchSource:0}: Error finding container e3c59306b82ea5223a1848b314404162d3337758f1f1eb395202afa94d2f337a: Status 404 returned error can't find the container with id e3c59306b82ea5223a1848b314404162d3337758f1f1eb395202afa94d2f337a Oct 31 03:26:53.355515 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:53.355406 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:53.998206 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:53.998171 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:54.037567 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:54.037532 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" event={"ID":"1712a46e-a034-40f6-a276-09a907dd082f","Type":"ContainerStarted","Data":"cd1a44801f0be21c518fb9f1e10bc9448b6c2057a17cc96369e9b39ffece50af"} Oct 31 03:26:54.037567 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:54.037572 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" event={"ID":"1712a46e-a034-40f6-a276-09a907dd082f","Type":"ContainerStarted","Data":"e3c59306b82ea5223a1848b314404162d3337758f1f1eb395202afa94d2f337a"} Oct 31 03:26:54.056423 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:54.056370 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podStartSLOduration=2.056357532 podStartE2EDuration="2.056357532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:54.054105369 +0000 UTC m=+648.588800326" watchObservedRunningTime="2026-10-31 03:26:54.056357532 +0000 UTC m=+648.591052487" Oct 31 03:26:54.704634 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:54.704582 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:54.714972 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:54.714938 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:26:55.402369 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:55.402304 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:55.748831 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:55.748800 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:55.761913 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:55.761885 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:56.066539 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:56.066457 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:26:56.085399 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:56.085352 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:56.834814 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:56.834773 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:57.408269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:57.408228 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:58.091864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:58.091826 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:58.783913 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:58.783871 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:59.008549 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.008500 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:59.544757 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.544724 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5"] Oct 31 03:26:59.551486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.551458 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:26:59.558790 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.558726 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-090ebd\"/\"openshift-service-ca.crt\"" Oct 31 03:26:59.558790 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.558769 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-090ebd\"/\"kube-root-ca.crt\"" Oct 31 03:26:59.559005 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.558929 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-090ebd\"/\"default-dockercfg-fnwtp\"" Oct 31 03:26:59.559621 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.559595 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5"] Oct 31 03:26:59.559785 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.559713 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-090ebd\"/\"premium-w3-090ebd-kserve-self-signed-certs\"" Oct 31 03:26:59.630299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.630265 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:59.661787 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.661752 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-model-cache\") pod \"premium-w3-090ebd-kserve-7dc669c77-s9qp5\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:26:59.661980 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.661803 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-dshm\") pod \"premium-w3-090ebd-kserve-7dc669c77-s9qp5\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:26:59.661980 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.661925 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k2rn\" (UniqueName: \"kubernetes.io/projected/1c3288c4-4250-4046-921b-4382c4d3ca93-kube-api-access-9k2rn\") pod \"premium-w3-090ebd-kserve-7dc669c77-s9qp5\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:26:59.661980 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.661955 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-home\") pod \"premium-w3-090ebd-kserve-7dc669c77-s9qp5\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:26:59.662160 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.662076 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c3288c4-4250-4046-921b-4382c4d3ca93-tls-certs\") pod \"premium-w3-090ebd-kserve-7dc669c77-s9qp5\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:26:59.662160 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.662135 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-tmp-dir\") pod \"premium-w3-090ebd-kserve-7dc669c77-s9qp5\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:26:59.763654 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.763620 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-home\") pod \"premium-w3-090ebd-kserve-7dc669c77-s9qp5\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:26:59.763895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.763740 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c3288c4-4250-4046-921b-4382c4d3ca93-tls-certs\") pod \"premium-w3-090ebd-kserve-7dc669c77-s9qp5\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:26:59.763895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.763773 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-tmp-dir\") pod \"premium-w3-090ebd-kserve-7dc669c77-s9qp5\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:26:59.764020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.763898 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-model-cache\") pod \"premium-w3-090ebd-kserve-7dc669c77-s9qp5\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:26:59.764020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.763938 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-dshm\") pod \"premium-w3-090ebd-kserve-7dc669c77-s9qp5\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:26:59.764133 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.764016 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k2rn\" (UniqueName: \"kubernetes.io/projected/1c3288c4-4250-4046-921b-4382c4d3ca93-kube-api-access-9k2rn\") pod \"premium-w3-090ebd-kserve-7dc669c77-s9qp5\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:26:59.764188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.764157 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-home\") pod \"premium-w3-090ebd-kserve-7dc669c77-s9qp5\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:26:59.764240 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.764207 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-tmp-dir\") pod \"premium-w3-090ebd-kserve-7dc669c77-s9qp5\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:26:59.764321 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.764293 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-model-cache\") pod \"premium-w3-090ebd-kserve-7dc669c77-s9qp5\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:26:59.766174 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.766145 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-dshm\") pod \"premium-w3-090ebd-kserve-7dc669c77-s9qp5\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:26:59.766467 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.766447 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c3288c4-4250-4046-921b-4382c4d3ca93-tls-certs\") pod \"premium-w3-090ebd-kserve-7dc669c77-s9qp5\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:26:59.772650 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.772622 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k2rn\" (UniqueName: \"kubernetes.io/projected/1c3288c4-4250-4046-921b-4382c4d3ca93-kube-api-access-9k2rn\") pod \"premium-w3-090ebd-kserve-7dc669c77-s9qp5\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:26:59.793548 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.793519 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:26:59.868894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:26:59.868819 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:27:00.144998 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:00.144968 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5"] Oct 31 03:27:00.146702 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:27:00.146642 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c3288c4_4250_4046_921b_4382c4d3ca93.slice/crio-68123b46158fd9ec0de8e612476c23f96b1b2e0b82abc78e38cb877a53646524 WatchSource:0}: Error finding container 68123b46158fd9ec0de8e612476c23f96b1b2e0b82abc78e38cb877a53646524: Status 404 returned error can't find the container with id 68123b46158fd9ec0de8e612476c23f96b1b2e0b82abc78e38cb877a53646524 Oct 31 03:27:00.475349 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:00.475316 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:01.086262 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:01.086221 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" event={"ID":"1c3288c4-4250-4046-921b-4382c4d3ca93","Type":"ContainerStarted","Data":"a2a288cfe831df10f02348c0b21102a6ec902dc025c0219d67fa72f5c78d9279"} Oct 31 03:27:01.086738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:01.086272 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" event={"ID":"1c3288c4-4250-4046-921b-4382c4d3ca93","Type":"ContainerStarted","Data":"68123b46158fd9ec0de8e612476c23f96b1b2e0b82abc78e38cb877a53646524"} Oct 31 03:27:01.113300 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:01.113231 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podStartSLOduration=2.113212625 podStartE2EDuration="2.113212625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:01.112518397 +0000 UTC m=+655.647213352" watchObservedRunningTime="2026-10-31 03:27:01.113212625 +0000 UTC m=+655.647907584" Oct 31 03:27:01.205069 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:01.205029 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:01.869210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:01.869178 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:27:01.883603 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:01.883558 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:02.580546 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:02.580508 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:02.714939 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:02.714899 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:27:02.727842 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:02.727814 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:27:03.111486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:03.111456 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:27:03.182542 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:03.182501 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:03.822979 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:03.822941 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:03.965352 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:03.965317 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:04.476443 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:04.476396 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:05.219838 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:05.219800 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:05.907370 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:05.907327 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:06.607152 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:06.607114 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:07.363666 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:07.363548 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:08.351910 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:08.351862 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:09.228600 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:09.228551 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:09.870152 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:09.870111 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:27:09.888555 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:09.888521 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:27:10.172154 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:10.172120 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:27:10.401506 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:10.401391 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:11.117433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:11.117386 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:11.908974 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:11.908936 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:12.130759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.130722 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc"] Oct 31 03:27:12.135888 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.135865 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:12.144080 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.144051 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f20dc4\"/\"kube-root-ca.crt\"" Oct 31 03:27:12.144080 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.144051 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-f20dc4\"/\"openshift-service-ca.crt\"" Oct 31 03:27:12.144299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.144130 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f20dc4\"/\"premium-w2-f20dc4-kserve-self-signed-certs\"" Oct 31 03:27:12.144299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.144198 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-f20dc4\"/\"default-dockercfg-5n7tw\"" Oct 31 03:27:12.145509 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.145486 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc"] Oct 31 03:27:12.221312 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.221232 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5vxf\" (UniqueName: \"kubernetes.io/projected/eca1025f-f016-4b3b-b32f-7ec9c69771b1-kube-api-access-m5vxf\") pod \"premium-w2-f20dc4-kserve-7c8544b957-dz7lc\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:12.221470 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.221377 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-home\") pod \"premium-w2-f20dc4-kserve-7c8544b957-dz7lc\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:12.221470 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.221432 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-dshm\") pod \"premium-w2-f20dc4-kserve-7c8544b957-dz7lc\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:12.221554 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.221527 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-model-cache\") pod \"premium-w2-f20dc4-kserve-7c8544b957-dz7lc\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:12.221590 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.221568 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-tmp-dir\") pod \"premium-w2-f20dc4-kserve-7c8544b957-dz7lc\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:12.221624 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.221593 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca1025f-f016-4b3b-b32f-7ec9c69771b1-tls-certs\") pod \"premium-w2-f20dc4-kserve-7c8544b957-dz7lc\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:12.237617 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.237582 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:12.323286 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.323248 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5vxf\" (UniqueName: \"kubernetes.io/projected/eca1025f-f016-4b3b-b32f-7ec9c69771b1-kube-api-access-m5vxf\") pod \"premium-w2-f20dc4-kserve-7c8544b957-dz7lc\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:12.323503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.323348 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-home\") pod \"premium-w2-f20dc4-kserve-7c8544b957-dz7lc\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:12.323503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.323386 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-dshm\") pod \"premium-w2-f20dc4-kserve-7c8544b957-dz7lc\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:12.323503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.323429 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-model-cache\") pod \"premium-w2-f20dc4-kserve-7c8544b957-dz7lc\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:12.323503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.323476 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-tmp-dir\") pod \"premium-w2-f20dc4-kserve-7c8544b957-dz7lc\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:12.323747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.323505 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca1025f-f016-4b3b-b32f-7ec9c69771b1-tls-certs\") pod \"premium-w2-f20dc4-kserve-7c8544b957-dz7lc\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:12.324971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.324941 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-model-cache\") pod \"premium-w2-f20dc4-kserve-7c8544b957-dz7lc\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:12.325212 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.325193 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-tmp-dir\") pod \"premium-w2-f20dc4-kserve-7c8544b957-dz7lc\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:12.325314 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.325192 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-home\") pod \"premium-w2-f20dc4-kserve-7c8544b957-dz7lc\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:12.326637 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.326611 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-dshm\") pod \"premium-w2-f20dc4-kserve-7c8544b957-dz7lc\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:12.327001 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.326981 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca1025f-f016-4b3b-b32f-7ec9c69771b1-tls-certs\") pod \"premium-w2-f20dc4-kserve-7c8544b957-dz7lc\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:12.330983 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.330958 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5vxf\" (UniqueName: \"kubernetes.io/projected/eca1025f-f016-4b3b-b32f-7ec9c69771b1-kube-api-access-m5vxf\") pod \"premium-w2-f20dc4-kserve-7c8544b957-dz7lc\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:12.451658 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.451623 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:12.737332 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.737302 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc"] Oct 31 03:27:12.739019 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:27:12.738987 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeca1025f_f016_4b3b_b32f_7ec9c69771b1.slice/crio-a7e0934b3a09de8dd3249ee00474637b93aec23c95364c852dbb7bc00cf416a1 WatchSource:0}: Error finding container a7e0934b3a09de8dd3249ee00474637b93aec23c95364c852dbb7bc00cf416a1: Status 404 returned error can't find the container with id a7e0934b3a09de8dd3249ee00474637b93aec23c95364c852dbb7bc00cf416a1 Oct 31 03:27:12.786809 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:12.786769 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:13.176472 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:13.176437 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" event={"ID":"eca1025f-f016-4b3b-b32f-7ec9c69771b1","Type":"ContainerStarted","Data":"6bef889339eba32ec308250696841174118b5d3a848c44c7fec7a22e93bba370"} Oct 31 03:27:13.176909 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:13.176486 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" event={"ID":"eca1025f-f016-4b3b-b32f-7ec9c69771b1","Type":"ContainerStarted","Data":"a7e0934b3a09de8dd3249ee00474637b93aec23c95364c852dbb7bc00cf416a1"} Oct 31 03:27:13.196736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:13.196666 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podStartSLOduration=1.196652511 podStartE2EDuration="1.196652511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:13.195705756 +0000 UTC m=+667.730400712" watchObservedRunningTime="2026-10-31 03:27:13.196652511 +0000 UTC m=+667.731347465" Oct 31 03:27:13.452287 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:13.452200 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:13.483089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:13.483043 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:14.387490 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:14.387437 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:15.279293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:15.279233 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:15.509024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:15.508988 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:16.088135 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:16.088097 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:16.877078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:16.877041 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:17.127891 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:27:17.127804 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:27:17.793437 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:17.793400 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:18.763396 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:18.763351 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:19.230289 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.230253 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h"] Oct 31 03:27:19.234333 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.234314 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.250214 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.250179 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-bcc7a4-openshift-default-dockercfg-pmpv4\"" Oct 31 03:27:19.250365 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.250259 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h"] Oct 31 03:27:19.299971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.299925 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.300150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.300052 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.300150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.300104 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.300150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.300137 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.300262 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.300217 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.300262 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.300244 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.300324 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.300267 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.300324 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.300287 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbgvj\" (UniqueName: \"kubernetes.io/projected/dcfb42f0-a328-4179-9a5d-ac6669e132a8-kube-api-access-fbgvj\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.300389 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.300376 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.401309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.401275 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.401505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.401319 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.401505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.401353 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.401505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.401377 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.401505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.401400 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.401505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.401452 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.401505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.401498 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbgvj\" (UniqueName: \"kubernetes.io/projected/dcfb42f0-a328-4179-9a5d-ac6669e132a8-kube-api-access-fbgvj\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.401852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.401558 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.401852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.401710 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.401971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.401930 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.402023 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.401981 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.402073 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.402059 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.402179 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.402161 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.402246 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.402228 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.404306 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.404286 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.404435 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.404419 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.413414 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.413385 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.413536 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.413495 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbgvj\" (UniqueName: \"kubernetes.io/projected/dcfb42f0-a328-4179-9a5d-ac6669e132a8-kube-api-access-fbgvj\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.550967 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.550883 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:19.581195 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.581157 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:19.848376 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:27:19.848342 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcfb42f0_a328_4179_9a5d_ac6669e132a8.slice/crio-0edb14b34644c09a53b2df5a24c57640503edec91fdb6ea372ab1b7a8194d606 WatchSource:0}: Error finding container 0edb14b34644c09a53b2df5a24c57640503edec91fdb6ea372ab1b7a8194d606: Status 404 returned error can't find the container with id 0edb14b34644c09a53b2df5a24c57640503edec91fdb6ea372ab1b7a8194d606 Oct 31 03:27:19.848884 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.848431 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h"] Oct 31 03:27:19.851387 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.851348 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:27:19.851511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.851439 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:27:19.851511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:19.851485 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:27:20.229293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:20.229252 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" event={"ID":"dcfb42f0-a328-4179-9a5d-ac6669e132a8","Type":"ContainerStarted","Data":"2a9717e9f16ee89232fadcc80299291331ed5ad5d1c8a5263458de8973929d4a"} Oct 31 03:27:20.229585 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:20.229552 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" event={"ID":"dcfb42f0-a328-4179-9a5d-ac6669e132a8","Type":"ContainerStarted","Data":"0edb14b34644c09a53b2df5a24c57640503edec91fdb6ea372ab1b7a8194d606"} Oct 31 03:27:20.257151 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:20.257099 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" podStartSLOduration=1.257085356 podStartE2EDuration="1.257085356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:20.255861041 +0000 UTC m=+674.790556010" watchObservedRunningTime="2026-10-31 03:27:20.257085356 +0000 UTC m=+674.791780311" Oct 31 03:27:20.564758 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:20.553469 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:20.564758 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:20.553601 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:27:20.564758 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:20.553645 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" podUID="dcfb42f0-a328-4179-9a5d-ac6669e132a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 03:27:20.570345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:20.570312 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:21.497727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:21.495793 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:21.552140 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:21.552097 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:27:21.552295 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:21.552174 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" podUID="dcfb42f0-a328-4179-9a5d-ac6669e132a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 03:27:22.349701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:22.346194 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:22.453972 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:22.453911 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:22.469416 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:22.469378 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:22.485968 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:22.485930 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:22.553204 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:22.553013 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:27:22.553204 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:22.553064 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" podUID="dcfb42f0-a328-4179-9a5d-ac6669e132a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 03:27:23.220716 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:23.220654 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:23.286973 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:23.286930 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:27:23.552220 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:23.552120 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:27:23.552495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:23.552458 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" podUID="dcfb42f0-a328-4179-9a5d-ac6669e132a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 03:27:24.171166 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:24.171125 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:24.563757 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:24.560813 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:27:24.563757 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:24.560892 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" podUID="dcfb42f0-a328-4179-9a5d-ac6669e132a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 03:27:25.001648 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:25.001606 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:25.551406 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:25.551315 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 03:27:25.551406 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:25.551378 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" podUID="dcfb42f0-a328-4179-9a5d-ac6669e132a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 03:27:25.672334 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:25.672299 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:26.292578 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:26.292542 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:26.715335 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:26.715297 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:26.715987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:26.715964 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:26.717317 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:26.717293 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:27:27.006528 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:27.006375 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:27.947858 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:27.947821 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:28.806980 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:28.806937 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:29.399868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:29.399809 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:30.131096 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:30.131063 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:30.819354 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:30.819318 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:31.597504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:31.597467 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:32.519116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:32.519079 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:33.327913 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:33.327869 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:34.109700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.109117 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:34.618069 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.618034 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7"] Oct 31 03:27:34.623287 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.623262 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.630271 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.630247 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-7f3159-openshift-default-dockercfg-4mdls\"" Oct 31 03:27:34.635876 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.635844 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7"] Oct 31 03:27:34.725321 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.725285 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrmhl\" (UniqueName: \"kubernetes.io/projected/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-kube-api-access-rrmhl\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.725518 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.725445 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.725598 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.725551 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.725648 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.725594 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.725738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.725721 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.725825 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.725809 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.725883 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.725839 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.725932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.725913 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.725986 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.725960 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.827451 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.827410 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.827621 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.827465 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.827621 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.827494 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.827621 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.827535 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.827621 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.827588 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.827621 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.827615 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.827954 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.827645 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.827954 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.827752 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.827954 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.827809 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrmhl\" (UniqueName: \"kubernetes.io/projected/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-kube-api-access-rrmhl\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.827954 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.827907 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.828226 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.828171 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.828349 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.828327 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.828574 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.828546 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.828669 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.828596 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.830134 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.830114 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.830366 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.830349 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.836569 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.836548 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrmhl\" (UniqueName: \"kubernetes.io/projected/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-kube-api-access-rrmhl\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.836652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.836570 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:34.903435 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.903388 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:34.941738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:34.941666 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:35.236826 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:35.236344 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7"] Oct 31 03:27:35.244551 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:35.244511 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:27:35.244658 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:35.244586 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:27:35.244658 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:35.244613 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:27:35.378837 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:35.378798 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" event={"ID":"cdf49e86-d2b1-4888-8475-8a85dd69a6f3","Type":"ContainerStarted","Data":"40c306e87ba45cc24f8f8a23e301f90588e657aafd2470a31afc320248c53bca"} Oct 31 03:27:35.379007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:35.378867 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" event={"ID":"cdf49e86-d2b1-4888-8475-8a85dd69a6f3","Type":"ContainerStarted","Data":"5501d381cbe469ff57e524315ebf0c3609d9f285c811639ef46f62c2fd351c2b"} Oct 31 03:27:35.399502 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:35.399442 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" podStartSLOduration=1.399422163 podStartE2EDuration="1.399422163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:35.397910572 +0000 UTC m=+689.932605529" watchObservedRunningTime="2026-10-31 03:27:35.399422163 +0000 UTC m=+689.934117118" Oct 31 03:27:35.634320 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:35.634225 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:35.941849 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:35.941812 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:36.340761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:36.338822 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:36.949740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:36.947172 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:27:36.949740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:36.947239 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" podUID="cdf49e86-d2b1-4888-8475-8a85dd69a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:27:37.342766 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:37.338351 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:37.951474 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:37.949203 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:27:37.951474 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:37.949267 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" podUID="cdf49e86-d2b1-4888-8475-8a85dd69a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:27:38.567106 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:38.567072 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:38.947705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:38.944194 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:27:38.947705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:38.944286 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" podUID="cdf49e86-d2b1-4888-8475-8a85dd69a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:27:39.317389 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:39.317255 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:39.944197 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:39.944157 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:27:39.944933 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:39.944899 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" podUID="cdf49e86-d2b1-4888-8475-8a85dd69a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:27:40.001812 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:40.001762 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:40.711994 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:40.711369 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:40.945703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:40.945160 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:27:40.945703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:40.945239 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" podUID="cdf49e86-d2b1-4888-8475-8a85dd69a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:27:41.614418 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:41.613887 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:41.942515 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:41.942479 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 03:27:41.942741 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:41.942558 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" podUID="cdf49e86-d2b1-4888-8475-8a85dd69a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 03:27:42.387016 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:42.386925 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:42.948355 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:42.948327 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:42.948652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:42.948630 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:42.949399 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:42.949382 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:27:43.117300 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:43.117264 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:43.820923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:43.820884 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:44.818134 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:44.818098 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:45.545338 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:45.545304 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:46.524343 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:46.524306 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:46.916472 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:46.916433 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:47.129916 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:27:47.129879 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:27:47.592484 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:47.592446 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:48.309941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:48.309901 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:49.069976 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:49.069939 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:49.682194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:49.682148 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:50.285970 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:50.285934 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:50.994213 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:50.994175 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:51.604965 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:51.604907 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:52.403823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:52.403785 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:53.198578 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:53.198543 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:53.799410 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:53.799368 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:54.497118 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:54.497074 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:55.135668 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:55.135630 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:55.799639 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:55.797797 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:56.660117 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:56.660061 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:57.534257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:57.534223 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:58.507457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:58.507407 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:59.288915 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:59.288873 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:27:59.995607 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:27:59.995563 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:00.678231 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:00.678184 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:01.348813 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:01.348777 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:02.301752 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:02.301699 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:02.901337 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:02.901272 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:03.580394 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:03.580355 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:04.003996 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.003943 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:04.310100 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.310009 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:04.538713 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.538657 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp"] Oct 31 03:28:04.543946 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.543923 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.549505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.549388 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-dda5cb-openshift-default-dockercfg-z7cvq\"" Oct 31 03:28:04.554727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.554698 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp"] Oct 31 03:28:04.622094 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.622003 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzvg5\" (UniqueName: \"kubernetes.io/projected/113276f1-a2bd-49bd-843e-22146959715e-kube-api-access-wzvg5\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.622094 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.622057 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113276f1-a2bd-49bd-843e-22146959715e-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.622289 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.622204 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.622289 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.622227 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113276f1-a2bd-49bd-843e-22146959715e-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.622408 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.622381 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113276f1-a2bd-49bd-843e-22146959715e-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.622738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.622531 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.622738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.622630 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.622738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.622668 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.623093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.622760 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.724167 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.724118 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzvg5\" (UniqueName: \"kubernetes.io/projected/113276f1-a2bd-49bd-843e-22146959715e-kube-api-access-wzvg5\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.724358 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.724178 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113276f1-a2bd-49bd-843e-22146959715e-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.724358 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.724207 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.724358 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.724224 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113276f1-a2bd-49bd-843e-22146959715e-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.724358 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.724293 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113276f1-a2bd-49bd-843e-22146959715e-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.724358 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.724354 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.724647 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.724425 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.724647 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.724464 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.724647 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.724493 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.724839 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.724813 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.724902 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.724837 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.724959 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.724937 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.725013 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.724992 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.725539 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.725504 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113276f1-a2bd-49bd-843e-22146959715e-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.726641 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.726622 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.727270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.727242 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113276f1-a2bd-49bd-843e-22146959715e-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.735213 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.735186 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113276f1-a2bd-49bd-843e-22146959715e-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.735397 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.735373 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzvg5\" (UniqueName: \"kubernetes.io/projected/113276f1-a2bd-49bd-843e-22146959715e-kube-api-access-wzvg5\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:04.860893 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:04.860862 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:05.147630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:05.147602 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp"] Oct 31 03:28:05.149472 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:28:05.149441 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod113276f1_a2bd_49bd_843e_22146959715e.slice/crio-eb0db1e46d38c473025d618730c48154049cc7f8791a95028f5377dd1cd40c0f WatchSource:0}: Error finding container eb0db1e46d38c473025d618730c48154049cc7f8791a95028f5377dd1cd40c0f: Status 404 returned error can't find the container with id eb0db1e46d38c473025d618730c48154049cc7f8791a95028f5377dd1cd40c0f Oct 31 03:28:05.152081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:05.152048 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:28:05.152197 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:05.152112 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:28:05.152197 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:05.152141 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:28:05.289724 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:05.287476 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:05.641291 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:05.641232 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" event={"ID":"113276f1-a2bd-49bd-843e-22146959715e","Type":"ContainerStarted","Data":"d5b262180510f1e0c68cc21a4af530afa514fd662ce4ea7ccd9386abef0405e9"} Oct 31 03:28:05.641454 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:05.641316 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" event={"ID":"113276f1-a2bd-49bd-843e-22146959715e","Type":"ContainerStarted","Data":"eb0db1e46d38c473025d618730c48154049cc7f8791a95028f5377dd1cd40c0f"} Oct 31 03:28:05.667412 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:05.667353 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" podStartSLOduration=1.667335467 podStartE2EDuration="1.667335467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:05.664290867 +0000 UTC m=+720.198985823" watchObservedRunningTime="2026-10-31 03:28:05.667335467 +0000 UTC m=+720.202030421" Oct 31 03:28:05.861945 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:05.861886 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:05.920945 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:05.920335 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:06.624914 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:06.624882 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:06.863633 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:06.863599 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:28:06.863827 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:06.863655 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" podUID="113276f1-a2bd-49bd-843e-22146959715e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 03:28:07.325096 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:07.324989 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:07.866154 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:07.866116 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:28:07.866787 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:07.866750 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" podUID="113276f1-a2bd-49bd-843e-22146959715e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 03:28:08.326541 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:08.326504 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:08.864175 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:08.864131 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:28:08.864531 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:08.864494 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" podUID="113276f1-a2bd-49bd-843e-22146959715e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 03:28:09.865816 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:09.865777 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:28:09.866420 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:09.866389 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" podUID="113276f1-a2bd-49bd-843e-22146959715e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 03:28:10.108889 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:10.108834 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:28:10.968905 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:10.968867 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:28:10.969450 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:10.969418 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" podUID="113276f1-a2bd-49bd-843e-22146959715e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 03:28:11.064700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:11.063378 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:28:11.111698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:11.109806 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:28:11.111698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:11.110737 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:28:11.188945 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:11.188887 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:28:11.202455 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:11.199404 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:11.343965 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:11.343867 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:28:11.870151 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:11.870098 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:28:11.870485 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:11.870449 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" podUID="113276f1-a2bd-49bd-843e-22146959715e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 03:28:12.079702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:12.065252 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:28:12.079702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:12.065697 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:28:12.079702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:12.068101 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:28:12.313542 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:12.313508 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:28:12.861894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:12.861862 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 03:28:12.862193 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:12.862164 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" podUID="113276f1-a2bd-49bd-843e-22146959715e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 03:28:12.863003 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:12.862974 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:13.607576 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:13.607518 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:13.866730 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:13.866633 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:13.866969 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:13.866950 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:13.867779 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:13.867758 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:28:14.162282 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:14.162247 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:14.391150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:14.391111 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:15.016568 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:15.016531 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:15.619786 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:15.619749 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:15.842247 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:15.842210 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:16.516595 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:16.516556 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:17.130910 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:28:17.130873 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:28:17.392961 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:17.392860 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:18.209945 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:18.209900 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:19.112010 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:19.111972 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:19.896105 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:19.896056 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:20.765816 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:20.765764 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:21.448899 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:21.448857 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:22.086818 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:22.086781 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:22.905268 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:22.905229 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:23.600500 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:23.600451 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:23.886709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:23.886585 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:24.393190 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:24.393147 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:25.090873 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:25.090837 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:25.792071 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:25.792026 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:26.482104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:26.482064 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:27.192901 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:27.192671 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:27.905566 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:27.905527 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:28.774819 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:28.774777 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:29.071270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.071189 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr"] Oct 31 03:28:29.075496 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.075476 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:29.089174 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.089145 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr"] Oct 31 03:28:29.095110 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.095084 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-090ebd\"/\"distinct-w3-090ebd-kserve-self-signed-certs\"" Oct 31 03:28:29.142248 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.142211 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:29.169310 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.169278 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-model-cache\") pod \"distinct-w3-090ebd-kserve-f4dd967c6-tslkr\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:29.169522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.169321 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqqbt\" (UniqueName: \"kubernetes.io/projected/6c2b41fc-ff94-4a75-affd-81795799528d-kube-api-access-mqqbt\") pod \"distinct-w3-090ebd-kserve-f4dd967c6-tslkr\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:29.169522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.169442 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c2b41fc-ff94-4a75-affd-81795799528d-tls-certs\") pod \"distinct-w3-090ebd-kserve-f4dd967c6-tslkr\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:29.169522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.169502 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-dshm\") pod \"distinct-w3-090ebd-kserve-f4dd967c6-tslkr\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:29.169731 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.169583 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-tmp-dir\") pod \"distinct-w3-090ebd-kserve-f4dd967c6-tslkr\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:29.169731 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.169705 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-home\") pod \"distinct-w3-090ebd-kserve-f4dd967c6-tslkr\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:29.270513 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.270480 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-home\") pod \"distinct-w3-090ebd-kserve-f4dd967c6-tslkr\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:29.270735 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.270586 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-model-cache\") pod \"distinct-w3-090ebd-kserve-f4dd967c6-tslkr\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:29.270735 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.270613 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqqbt\" (UniqueName: \"kubernetes.io/projected/6c2b41fc-ff94-4a75-affd-81795799528d-kube-api-access-mqqbt\") pod \"distinct-w3-090ebd-kserve-f4dd967c6-tslkr\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:29.270735 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.270693 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c2b41fc-ff94-4a75-affd-81795799528d-tls-certs\") pod \"distinct-w3-090ebd-kserve-f4dd967c6-tslkr\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:29.270982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.270758 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-dshm\") pod \"distinct-w3-090ebd-kserve-f4dd967c6-tslkr\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:29.270982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.270774 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-tmp-dir\") pod \"distinct-w3-090ebd-kserve-f4dd967c6-tslkr\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:29.271097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.271019 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-home\") pod \"distinct-w3-090ebd-kserve-f4dd967c6-tslkr\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:29.271195 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.271172 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-tmp-dir\") pod \"distinct-w3-090ebd-kserve-f4dd967c6-tslkr\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:29.271271 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.271188 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-model-cache\") pod \"distinct-w3-090ebd-kserve-f4dd967c6-tslkr\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:29.273481 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.273456 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-dshm\") pod \"distinct-w3-090ebd-kserve-f4dd967c6-tslkr\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:29.273481 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.273475 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c2b41fc-ff94-4a75-affd-81795799528d-tls-certs\") pod \"distinct-w3-090ebd-kserve-f4dd967c6-tslkr\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:29.278373 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.278353 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqqbt\" (UniqueName: \"kubernetes.io/projected/6c2b41fc-ff94-4a75-affd-81795799528d-kube-api-access-mqqbt\") pod \"distinct-w3-090ebd-kserve-f4dd967c6-tslkr\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:29.394472 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.394393 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:29.502623 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.502582 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:29.679898 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:29.679875 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr"] Oct 31 03:28:29.680950 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:28:29.680927 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c2b41fc_ff94_4a75_affd_81795799528d.slice/crio-3a35650bb6112ba2ffa1c03456c4ed9888e54fb6aed498199bc317efc012d242 WatchSource:0}: Error finding container 3a35650bb6112ba2ffa1c03456c4ed9888e54fb6aed498199bc317efc012d242: Status 404 returned error can't find the container with id 3a35650bb6112ba2ffa1c03456c4ed9888e54fb6aed498199bc317efc012d242 Oct 31 03:28:30.165925 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:30.165892 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" event={"ID":"6c2b41fc-ff94-4a75-affd-81795799528d","Type":"ContainerStarted","Data":"8e0af768643439fb9460606bab9bbb76de89707ee5f82da82ccdc4a365b2c9c4"} Oct 31 03:28:30.166343 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:30.165939 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" event={"ID":"6c2b41fc-ff94-4a75-affd-81795799528d","Type":"ContainerStarted","Data":"3a35650bb6112ba2ffa1c03456c4ed9888e54fb6aed498199bc317efc012d242"} Oct 31 03:28:30.187013 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:30.186961 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podStartSLOduration=1.186948736 podStartE2EDuration="1.186948736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:30.183064697 +0000 UTC m=+744.717759652" watchObservedRunningTime="2026-10-31 03:28:30.186948736 +0000 UTC m=+744.721643691" Oct 31 03:28:30.395449 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:30.395414 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:30.403222 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:30.403185 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:31.098249 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:31.098147 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:31.780937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:31.780898 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:32.684698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:32.684217 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:33.629737 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.627194 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s"] Oct 31 03:28:33.637090 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.637060 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.648275 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.648242 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-090ebd-openshift-default-dockercfg-kj8g4\"" Oct 31 03:28:33.662934 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.662891 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s"] Oct 31 03:28:33.789645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.757981 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.789645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.786875 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br78q\" (UniqueName: \"kubernetes.io/projected/7658fa72-7a26-4904-b702-6006f741af65-kube-api-access-br78q\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.789645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.787072 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7658fa72-7a26-4904-b702-6006f741af65-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.789645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.787157 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.789645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.787207 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.789645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.787231 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.789645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.787297 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7658fa72-7a26-4904-b702-6006f741af65-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.789645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.787347 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7658fa72-7a26-4904-b702-6006f741af65-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.789645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.787429 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.889301 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.889202 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.889469 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.889310 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.889469 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.889378 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br78q\" (UniqueName: \"kubernetes.io/projected/7658fa72-7a26-4904-b702-6006f741af65-kube-api-access-br78q\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.889612 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.889483 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7658fa72-7a26-4904-b702-6006f741af65-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.889612 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.889541 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.889612 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.889575 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.889612 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.889602 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.889867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.889646 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7658fa72-7a26-4904-b702-6006f741af65-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.889867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.889724 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7658fa72-7a26-4904-b702-6006f741af65-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.890632 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.890602 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7658fa72-7a26-4904-b702-6006f741af65-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.890965 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.890941 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.891271 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.891250 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.892412 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.892387 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.892780 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.892758 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.894986 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.894964 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.895289 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.895263 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7658fa72-7a26-4904-b702-6006f741af65-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.900038 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.900016 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7658fa72-7a26-4904-b702-6006f741af65-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.900491 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.900469 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br78q\" (UniqueName: \"kubernetes.io/projected/7658fa72-7a26-4904-b702-6006f741af65-kube-api-access-br78q\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:33.974555 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:33.974512 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:34.226747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:34.226707 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:34.267723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:34.267697 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s"] Oct 31 03:28:34.269775 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:28:34.269746 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7658fa72_7a26_4904_b702_6006f741af65.slice/crio-a4346efeefce4787f837ae6ab78acd2b722abeb04f65651b27fba06761d81a56 WatchSource:0}: Error finding container a4346efeefce4787f837ae6ab78acd2b722abeb04f65651b27fba06761d81a56: Status 404 returned error can't find the container with id a4346efeefce4787f837ae6ab78acd2b722abeb04f65651b27fba06761d81a56 Oct 31 03:28:34.272341 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:34.272309 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:28:34.272420 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:34.272372 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:28:34.272420 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:34.272402 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:28:34.334769 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:34.334735 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" event={"ID":"7658fa72-7a26-4904-b702-6006f741af65","Type":"ContainerStarted","Data":"a4346efeefce4787f837ae6ab78acd2b722abeb04f65651b27fba06761d81a56"} Oct 31 03:28:34.973198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:34.973163 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:35.359700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:35.359580 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" event={"ID":"7658fa72-7a26-4904-b702-6006f741af65","Type":"ContainerStarted","Data":"a08686043fc4d038cd9d6b8bc4697bb97c7a49f47f0a4cb09fcb72e101d0ffe9"} Oct 31 03:28:35.391063 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:35.391009 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" podStartSLOduration=2.390992478 podStartE2EDuration="2.390992478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:35.390485865 +0000 UTC m=+749.925180820" watchObservedRunningTime="2026-10-31 03:28:35.390992478 +0000 UTC m=+749.925687432" Oct 31 03:28:35.791465 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:35.791428 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:35.974740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:35.974713 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:35.976667 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:35.976638 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:28:35.976831 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:35.976719 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" podUID="7658fa72-7a26-4904-b702-6006f741af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 03:28:36.556478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:36.554519 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:36.975261 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:36.975225 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:28:36.975727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:36.975279 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" podUID="7658fa72-7a26-4904-b702-6006f741af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 03:28:37.261418 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:37.261181 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:37.584670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:37.584577 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:37.974897 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:37.974852 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:28:37.975114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:37.974943 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" podUID="7658fa72-7a26-4904-b702-6006f741af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 03:28:38.051250 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:38.051210 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:38.954795 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:38.951447 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:38.975372 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:38.975336 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:28:38.975593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:38.975425 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" podUID="7658fa72-7a26-4904-b702-6006f741af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 03:28:39.394793 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:39.394705 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:39.407900 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:39.407875 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:39.801127 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:39.801084 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:39.975343 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:39.975301 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 03:28:39.975527 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:39.975373 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" podUID="7658fa72-7a26-4904-b702-6006f741af65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 03:28:40.407531 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:40.407500 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:28:40.496915 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:40.496866 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:41.048709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:41.045532 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:41.048709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:41.046394 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:41.048709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:41.047584 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:28:41.201470 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:41.201424 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:42.001229 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:42.001194 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:42.861881 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:42.860819 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:43.615850 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:43.615809 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:44.297276 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:44.297235 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:44.483498 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:44.483454 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:45.015473 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:45.015432 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:45.737619 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:45.737582 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:46.456110 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:46.456070 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:47.129712 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:28:47.129657 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:28:47.209901 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:47.209864 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:47.939993 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:47.939916 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:48.605090 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:48.605049 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:49.212336 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.212297 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87"] Oct 31 03:28:49.217929 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.217897 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.234883 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.234848 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87"] Oct 31 03:28:49.326747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.318653 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr7sj\" (UniqueName: \"kubernetes.io/projected/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-kube-api-access-nr7sj\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.326747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.318736 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.326747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.318884 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.326747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.318913 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.326747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.318995 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.326747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.319037 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.326747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.319070 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.326747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.319109 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.326747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.319138 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.422621 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.422575 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.423495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.423455 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.423640 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.423504 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.423640 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.423548 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.423640 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.423579 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.423846 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.423660 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr7sj\" (UniqueName: \"kubernetes.io/projected/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-kube-api-access-nr7sj\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.423846 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.423379 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.423846 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.423784 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.424001 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.423977 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.424056 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.424008 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.424137 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.424111 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.424349 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.424323 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.424450 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.424399 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.425554 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.425526 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.427239 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.427214 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.428076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.428057 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.434925 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.434895 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr7sj\" (UniqueName: \"kubernetes.io/projected/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-kube-api-access-nr7sj\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.435270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.435248 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.535398 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.535308 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:49.542214 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.542188 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:49.847395 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.846788 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87"] Oct 31 03:28:49.879157 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.857083 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:28:49.879157 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.857173 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:28:49.879157 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:49.857215 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:28:50.492056 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:50.492020 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" event={"ID":"b0b3f9b0-9f95-4556-87a2-0d7473db75f5","Type":"ContainerStarted","Data":"f19c8fb69294d695de002163e31d193eb96e5e1d4514f7950d3d3416c22d9302"} Oct 31 03:28:50.492633 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:50.492610 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" event={"ID":"b0b3f9b0-9f95-4556-87a2-0d7473db75f5","Type":"ContainerStarted","Data":"77caa69f9ea2072aeef64449e8ac6e300b1b18e25e5d41c2de15a8a717f48950"} Oct 31 03:28:50.540720 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:50.539044 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:50.889623 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:50.884826 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:51.542734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:51.541728 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 03:28:51.542734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:51.541909 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" podUID="b0b3f9b0-9f95-4556-87a2-0d7473db75f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 03:28:52.038742 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:52.033477 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:52.547701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:52.546115 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 03:28:52.547701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:52.546179 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" podUID="b0b3f9b0-9f95-4556-87a2-0d7473db75f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 03:28:52.872963 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:52.872870 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:52.973043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:52.972992 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" podStartSLOduration=3.972975575 podStartE2EDuration="3.972975575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:50.541152954 +0000 UTC m=+765.075847914" watchObservedRunningTime="2026-10-31 03:28:52.972975575 +0000 UTC m=+767.507670530" Oct 31 03:28:52.974170 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:52.974143 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6"] Oct 31 03:28:52.978403 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:52.978383 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:52.984958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:52.984924 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d86a77\"/\"kube-root-ca.crt\"" Oct 31 03:28:52.985556 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:52.985531 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d86a77\"/\"distinct-w0-d86a77-kserve-self-signed-certs\"" Oct 31 03:28:52.985925 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:52.985779 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d86a77\"/\"openshift-service-ca.crt\"" Oct 31 03:28:52.985925 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:52.985883 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d86a77\"/\"default-dockercfg-qfn76\"" Oct 31 03:28:52.989733 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:52.989713 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6"] Oct 31 03:28:53.061641 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.061609 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:53.093344 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.093308 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d61881b1-0191-43d7-9ac2-c060af4e8f6c-tls-certs\") pod \"distinct-w0-d86a77-kserve-85fb976dfd-xfxl6\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:53.093550 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.093374 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6psd\" (UniqueName: \"kubernetes.io/projected/d61881b1-0191-43d7-9ac2-c060af4e8f6c-kube-api-access-x6psd\") pod \"distinct-w0-d86a77-kserve-85fb976dfd-xfxl6\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:53.093630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.093603 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-model-cache\") pod \"distinct-w0-d86a77-kserve-85fb976dfd-xfxl6\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:53.093732 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.093699 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-tmp-dir\") pod \"distinct-w0-d86a77-kserve-85fb976dfd-xfxl6\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:53.093794 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.093754 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-home\") pod \"distinct-w0-d86a77-kserve-85fb976dfd-xfxl6\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:53.093794 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.093784 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-dshm\") pod \"distinct-w0-d86a77-kserve-85fb976dfd-xfxl6\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:53.194610 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.194574 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d61881b1-0191-43d7-9ac2-c060af4e8f6c-tls-certs\") pod \"distinct-w0-d86a77-kserve-85fb976dfd-xfxl6\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:53.194830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.194627 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6psd\" (UniqueName: \"kubernetes.io/projected/d61881b1-0191-43d7-9ac2-c060af4e8f6c-kube-api-access-x6psd\") pod \"distinct-w0-d86a77-kserve-85fb976dfd-xfxl6\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:53.194830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.194810 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-model-cache\") pod \"distinct-w0-d86a77-kserve-85fb976dfd-xfxl6\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:53.194916 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.194876 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-tmp-dir\") pod \"distinct-w0-d86a77-kserve-85fb976dfd-xfxl6\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:53.194951 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.194935 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-home\") pod \"distinct-w0-d86a77-kserve-85fb976dfd-xfxl6\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:53.195051 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.195024 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-dshm\") pod \"distinct-w0-d86a77-kserve-85fb976dfd-xfxl6\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:53.195212 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.195192 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-model-cache\") pod \"distinct-w0-d86a77-kserve-85fb976dfd-xfxl6\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:53.195317 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.195292 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-tmp-dir\") pod \"distinct-w0-d86a77-kserve-85fb976dfd-xfxl6\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:53.195383 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.195300 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-home\") pod \"distinct-w0-d86a77-kserve-85fb976dfd-xfxl6\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:53.197233 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.197214 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-dshm\") pod \"distinct-w0-d86a77-kserve-85fb976dfd-xfxl6\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:53.197433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.197413 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d61881b1-0191-43d7-9ac2-c060af4e8f6c-tls-certs\") pod \"distinct-w0-d86a77-kserve-85fb976dfd-xfxl6\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:53.202536 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.202505 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6psd\" (UniqueName: \"kubernetes.io/projected/d61881b1-0191-43d7-9ac2-c060af4e8f6c-kube-api-access-x6psd\") pod \"distinct-w0-d86a77-kserve-85fb976dfd-xfxl6\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:53.295351 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.295317 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:53.553668 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.550143 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 03:28:53.553668 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.550213 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" podUID="b0b3f9b0-9f95-4556-87a2-0d7473db75f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 03:28:53.606504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.606476 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6"] Oct 31 03:28:53.607756 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:28:53.607717 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd61881b1_0191_43d7_9ac2_c060af4e8f6c.slice/crio-1c31c8b11b46b4b62fac912133ffabe125c629f1efa9040cc5c0482adb4f9659 WatchSource:0}: Error finding container 1c31c8b11b46b4b62fac912133ffabe125c629f1efa9040cc5c0482adb4f9659: Status 404 returned error can't find the container with id 1c31c8b11b46b4b62fac912133ffabe125c629f1efa9040cc5c0482adb4f9659 Oct 31 03:28:53.798472 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:53.797998 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:54.551327 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:54.551290 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 03:28:54.551595 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:54.551565 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" podUID="b0b3f9b0-9f95-4556-87a2-0d7473db75f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 03:28:54.554635 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:54.554601 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" event={"ID":"d61881b1-0191-43d7-9ac2-c060af4e8f6c","Type":"ContainerStarted","Data":"f91006e5ba6533082e0b81908d537ffce86d3551192c5cf45e9bbaf2299a8046"} Oct 31 03:28:54.555011 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:54.554641 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" event={"ID":"d61881b1-0191-43d7-9ac2-c060af4e8f6c","Type":"ContainerStarted","Data":"1c31c8b11b46b4b62fac912133ffabe125c629f1efa9040cc5c0482adb4f9659"} Oct 31 03:28:54.572498 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:54.572452 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podStartSLOduration=2.572439034 podStartE2EDuration="2.572439034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:54.569923113 +0000 UTC m=+769.104618066" watchObservedRunningTime="2026-10-31 03:28:54.572439034 +0000 UTC m=+769.107134013" Oct 31 03:28:54.681972 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:54.681932 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:55.296075 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:55.296043 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:28:55.396946 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:55.396911 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:55.543701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:55.542111 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 03:28:55.543701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:55.542179 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" podUID="b0b3f9b0-9f95-4556-87a2-0d7473db75f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 03:28:56.160399 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:56.160324 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:56.542738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:56.542187 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 03:28:56.542738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:56.542255 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" podUID="b0b3f9b0-9f95-4556-87a2-0d7473db75f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 03:28:56.783075 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:56.783036 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:56.913726 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:56.913242 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:57.543905 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:57.543869 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:57.544705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:57.544656 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:57.545737 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:57.545715 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:28:57.760391 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:57.760308 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:58.508748 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:58.508708 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:28:59.326035 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:28:59.325996 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:00.087037 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.087002 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:00.269734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.269695 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw"] Oct 31 03:29:00.275401 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.275377 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:00.280224 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.280196 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-090ebd\"/\"distinct-2-w3-090ebd-kserve-self-signed-certs\"" Oct 31 03:29:00.282943 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.282917 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw"] Oct 31 03:29:00.368955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.368866 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:00.410278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.410225 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-dshm\") pod \"distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:00.410495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.410311 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-home\") pod \"distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:00.410495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.410419 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-model-cache\") pod \"distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:00.410495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.410454 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-tls-certs\") pod \"distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:00.410661 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.410496 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-tmp-dir\") pod \"distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:00.410661 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.410598 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq45f\" (UniqueName: \"kubernetes.io/projected/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-kube-api-access-vq45f\") pod \"distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:00.512218 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.512172 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-dshm\") pod \"distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:00.512436 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.512268 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-home\") pod \"distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:00.512436 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.512391 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-model-cache\") pod \"distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:00.512436 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.512424 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-tls-certs\") pod \"distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:00.512602 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.512558 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-tmp-dir\") pod \"distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:00.512664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.512619 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq45f\" (UniqueName: \"kubernetes.io/projected/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-kube-api-access-vq45f\") pod \"distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:00.512769 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.512712 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-home\") pod \"distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:00.512825 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.512785 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-model-cache\") pod \"distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:00.512925 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.512900 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-tmp-dir\") pod \"distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:00.514654 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.514635 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-dshm\") pod \"distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:00.514825 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.514806 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-tls-certs\") pod \"distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:00.521364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.521342 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq45f\" (UniqueName: \"kubernetes.io/projected/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-kube-api-access-vq45f\") pod \"distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:00.592938 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.592902 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:00.845868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.845827 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:00.886850 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:00.886821 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw"] Oct 31 03:29:00.887813 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:29:00.887786 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b771958_a3d1_40a9_aa74_81b7b33aa1e5.slice/crio-7d36a300e5d4e2f7aeb96870f1ad3a66e8793b96b403219ac2ee2979052963fc WatchSource:0}: Error finding container 7d36a300e5d4e2f7aeb96870f1ad3a66e8793b96b403219ac2ee2979052963fc: Status 404 returned error can't find the container with id 7d36a300e5d4e2f7aeb96870f1ad3a66e8793b96b403219ac2ee2979052963fc Oct 31 03:29:01.600710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:01.600643 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:01.611958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:01.611923 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" event={"ID":"2b771958-a3d1-40a9-aa74-81b7b33aa1e5","Type":"ContainerStarted","Data":"6ea2af8e4e6ca860f8372354edb18735327dc836ff1970eed9959a69085ac28c"} Oct 31 03:29:01.612117 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:01.611972 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" event={"ID":"2b771958-a3d1-40a9-aa74-81b7b33aa1e5","Type":"ContainerStarted","Data":"7d36a300e5d4e2f7aeb96870f1ad3a66e8793b96b403219ac2ee2979052963fc"} Oct 31 03:29:01.633067 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:01.633010 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podStartSLOduration=1.63299753 podStartE2EDuration="1.63299753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:01.631057404 +0000 UTC m=+776.165752358" watchObservedRunningTime="2026-10-31 03:29:01.63299753 +0000 UTC m=+776.167692484" Oct 31 03:29:02.368099 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:02.368060 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:02.593515 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:02.593474 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:03.121089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:03.121054 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:03.295665 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:03.295632 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:29:03.310004 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:03.309974 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:29:03.461093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:03.461056 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:03.637989 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:03.637952 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:29:03.801224 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:03.801140 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:04.387131 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.387099 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b"] Oct 31 03:29:04.391412 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.391389 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.405106 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.405079 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b"] Oct 31 03:29:04.460579 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.460544 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.460788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.460636 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.460788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.460706 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.460909 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.460861 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.460959 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.460914 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.461004 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.460984 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.461039 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.461022 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.461102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.461091 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.461136 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.461117 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttthq\" (UniqueName: \"kubernetes.io/projected/6747ac36-d4a2-4a74-a97c-2c53a909da9d-kube-api-access-ttthq\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.486911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.486863 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:04.561841 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.561805 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.562024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.561909 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.562024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.561952 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.562024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.561975 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.562024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.562011 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.562237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.562078 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.562237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.562107 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttthq\" (UniqueName: \"kubernetes.io/projected/6747ac36-d4a2-4a74-a97c-2c53a909da9d-kube-api-access-ttthq\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.562237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.562176 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.562237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.562231 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.562443 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.562354 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.562496 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.562436 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.562642 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.562619 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.562710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.562628 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.562835 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.562817 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.565050 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.564995 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.565172 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.565127 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.570279 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.570258 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttthq\" (UniqueName: \"kubernetes.io/projected/6747ac36-d4a2-4a74-a97c-2c53a909da9d-kube-api-access-ttthq\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.570366 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.570293 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:04.707310 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:04.707276 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:05.005311 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:05.005270 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b"] Oct 31 03:29:05.008518 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:29:05.008464 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6747ac36_d4a2_4a74_a97c_2c53a909da9d.slice/crio-c873da69502e6c79b1cfe9dfdb6ca621ee2a5b7386ec1c63ee1e681e71934d34 WatchSource:0}: Error finding container c873da69502e6c79b1cfe9dfdb6ca621ee2a5b7386ec1c63ee1e681e71934d34: Status 404 returned error can't find the container with id c873da69502e6c79b1cfe9dfdb6ca621ee2a5b7386ec1c63ee1e681e71934d34 Oct 31 03:29:05.012844 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:05.012806 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:05.012955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:05.012875 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:05.012955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:05.012902 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:05.157255 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:05.154280 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:05.642550 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:05.642512 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" event={"ID":"6747ac36-d4a2-4a74-a97c-2c53a909da9d","Type":"ContainerStarted","Data":"384285a7d4ba0dd08417683c21bbdf4b4d2ff420e9c2f12ae2698493c96d8196"} Oct 31 03:29:05.643080 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:05.642571 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" event={"ID":"6747ac36-d4a2-4a74-a97c-2c53a909da9d","Type":"ContainerStarted","Data":"c873da69502e6c79b1cfe9dfdb6ca621ee2a5b7386ec1c63ee1e681e71934d34"} Oct 31 03:29:05.664982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:05.664924 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" podStartSLOduration=1.66491006 podStartE2EDuration="1.66491006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:05.661946796 +0000 UTC m=+780.196641757" watchObservedRunningTime="2026-10-31 03:29:05.66491006 +0000 UTC m=+780.199605014" Oct 31 03:29:05.711703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:05.708349 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:05.789049 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:05.789013 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:06.521813 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:06.521771 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:06.709661 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:06.709623 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:29:06.710118 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:06.709715 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" podUID="6747ac36-d4a2-4a74-a97c-2c53a909da9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 03:29:07.399486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:07.399393 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:07.707668 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:07.707632 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:29:07.707851 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:07.707697 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" podUID="6747ac36-d4a2-4a74-a97c-2c53a909da9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 03:29:08.105433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:08.105343 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:08.708695 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:08.708648 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:29:08.708887 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:08.708739 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" podUID="6747ac36-d4a2-4a74-a97c-2c53a909da9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 03:29:08.941115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:08.941074 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:09.708694 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:09.708648 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:29:09.709103 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:09.708716 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" podUID="6747ac36-d4a2-4a74-a97c-2c53a909da9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 03:29:09.837218 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:09.837176 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:10.600786 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:10.600745 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:10.624713 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:10.624660 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:10.708530 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:10.708114 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:29:10.708530 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:10.708175 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" podUID="6747ac36-d4a2-4a74-a97c-2c53a909da9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 03:29:10.912281 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:10.912248 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:11.167883 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:11.163318 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:11.730727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:11.724236 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 03:29:11.730727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:11.724311 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" podUID="6747ac36-d4a2-4a74-a97c-2c53a909da9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 03:29:12.244698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:12.243799 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:12.260702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:12.245309 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:29:12.297724 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:12.297600 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:12.774698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:12.774330 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:13.052754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:13.051028 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:13.052754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:13.052717 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:13.056699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:13.055897 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:29:13.900036 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:13.900003 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:14.095135 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:14.095094 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:15.739873 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:15.739822 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:15.876249 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:15.876209 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:16.400654 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:16.400611 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:17.124933 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:17.123459 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:17.140150 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:29:17.140113 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:29:18.256925 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.256880 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:18.658805 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.658761 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2"] Oct 31 03:29:18.665175 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.665146 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.675725 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.675696 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2"] Oct 31 03:29:18.771136 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.771094 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.771136 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.771139 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l89tp\" (UniqueName: \"kubernetes.io/projected/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-kube-api-access-l89tp\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.771406 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.771209 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.771406 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.771233 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.771406 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.771385 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.771535 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.771462 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.771535 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.771504 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.771687 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.771660 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.771743 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.771712 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.872730 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.872659 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.872937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.872763 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.872937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.872786 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l89tp\" (UniqueName: \"kubernetes.io/projected/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-kube-api-access-l89tp\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.872937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.872834 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.872937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.872872 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.872937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.872916 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.873209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.872962 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.873209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.873000 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.873209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.873181 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.873209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.873189 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.873412 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.873344 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.873475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.873445 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.873552 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.873530 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.873667 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.873642 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.875384 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.875366 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.875435 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.875391 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.881310 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.881282 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l89tp\" (UniqueName: \"kubernetes.io/projected/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-kube-api-access-l89tp\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.881614 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.881592 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:18.983378 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:18.983287 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:19.196402 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.196364 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:19.274789 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.274755 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2"] Oct 31 03:29:19.275652 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:29:19.275615 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3630f1_9fe3_4a38_8095_2dba83b4e1a7.slice/crio-9de1bc52a1f035a963b7bddbe24574a2cc62c28f7c1bbc613fad80e6d2623c16 WatchSource:0}: Error finding container 9de1bc52a1f035a963b7bddbe24574a2cc62c28f7c1bbc613fad80e6d2623c16: Status 404 returned error can't find the container with id 9de1bc52a1f035a963b7bddbe24574a2cc62c28f7c1bbc613fad80e6d2623c16 Oct 31 03:29:19.278297 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.278266 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:19.278367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.278333 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:19.278367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.278364 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:19.770338 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.767289 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r"] Oct 31 03:29:19.773988 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.773961 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.782080 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.782049 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r"] Oct 31 03:29:19.866710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.866653 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" event={"ID":"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7","Type":"ContainerStarted","Data":"3d9e5f438cd0c3ceb84a8f79ab1e58deb2a516f537763689557990c38223044b"} Oct 31 03:29:19.866887 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.866737 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" event={"ID":"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7","Type":"ContainerStarted","Data":"9de1bc52a1f035a963b7bddbe24574a2cc62c28f7c1bbc613fad80e6d2623c16"} Oct 31 03:29:19.886823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.886795 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtjxw\" (UniqueName: \"kubernetes.io/projected/60771f19-56cc-40a1-a27c-b09f23ac9903-kube-api-access-qtjxw\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.886998 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.886840 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.886998 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.886932 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.887102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.887002 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" podStartSLOduration=1.8869852360000001 podStartE2EDuration="1.886985236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:19.886494324 +0000 UTC m=+794.421189281" watchObservedRunningTime="2026-10-31 03:29:19.886985236 +0000 UTC m=+794.421680193" Oct 31 03:29:19.887102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.887064 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.887209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.887188 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.887268 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.887220 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60771f19-56cc-40a1-a27c-b09f23ac9903-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.887268 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.887247 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.887379 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.887353 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.887489 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.887472 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.983887 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.983848 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:19.989244 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.989217 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.989382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.989284 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.989382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.989307 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60771f19-56cc-40a1-a27c-b09f23ac9903-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.989382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.989330 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.989382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.989360 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.989568 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.989463 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.989731 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.989705 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.989837 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.989798 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.989997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.989973 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtjxw\" (UniqueName: \"kubernetes.io/projected/60771f19-56cc-40a1-a27c-b09f23ac9903-kube-api-access-qtjxw\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.990092 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.990072 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.990166 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.990139 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60771f19-56cc-40a1-a27c-b09f23ac9903-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.990231 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.990190 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.990290 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.990247 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.990485 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.990462 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.991809 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.991780 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:19.992030 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:19.992010 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:20.000794 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:20.000773 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:20.001011 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:20.000994 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtjxw\" (UniqueName: \"kubernetes.io/projected/60771f19-56cc-40a1-a27c-b09f23ac9903-kube-api-access-qtjxw\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:20.096758 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:20.096643 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:20.102875 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:20.102851 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:20.456065 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:20.455727 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r"] Oct 31 03:29:20.458124 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:29:20.458090 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60771f19_56cc_40a1_a27c_b09f23ac9903.slice/crio-6a5f9c597dd5243adf72fdaae021da328cf87023f8b76a0370586f17617f7db1 WatchSource:0}: Error finding container 6a5f9c597dd5243adf72fdaae021da328cf87023f8b76a0370586f17617f7db1: Status 404 returned error can't find the container with id 6a5f9c597dd5243adf72fdaae021da328cf87023f8b76a0370586f17617f7db1 Oct 31 03:29:20.460823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:20.460790 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:20.460911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:20.460860 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:20.460911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:20.460888 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:20.875847 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:20.875749 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" event={"ID":"60771f19-56cc-40a1-a27c-b09f23ac9903","Type":"ContainerStarted","Data":"78c8fca1454169c1ff41d6bc7cb33be6114c658e21bbdff35bacfc579817faa1"} Oct 31 03:29:20.875847 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:20.875816 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" event={"ID":"60771f19-56cc-40a1-a27c-b09f23ac9903","Type":"ContainerStarted","Data":"6a5f9c597dd5243adf72fdaae021da328cf87023f8b76a0370586f17617f7db1"} Oct 31 03:29:20.958848 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:20.958783 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" podStartSLOduration=1.95876706 podStartE2EDuration="1.95876706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:20.94816673 +0000 UTC m=+795.482861699" watchObservedRunningTime="2026-10-31 03:29:20.95876706 +0000 UTC m=+795.493462015" Oct 31 03:29:20.985268 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:20.985230 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:29:20.985445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:20.985292 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" podUID="cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 03:29:21.104244 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:21.104210 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:21.106175 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:21.106146 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 03:29:21.106315 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:21.106195 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" podUID="60771f19-56cc-40a1-a27c-b09f23ac9903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 03:29:21.168496 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:21.167178 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:21.984326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:21.984272 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:29:21.984326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:21.984327 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" podUID="cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 03:29:22.001138 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:22.001108 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:22.104241 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:22.104210 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 03:29:22.104404 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:22.104260 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" podUID="60771f19-56cc-40a1-a27c-b09f23ac9903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 03:29:22.714306 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:22.714269 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:22.857703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:22.853158 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:22.984298 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:22.984200 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:29:22.984298 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:22.984259 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" podUID="cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 03:29:23.103720 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:23.103663 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 03:29:23.103878 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:23.103744 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" podUID="60771f19-56cc-40a1-a27c-b09f23ac9903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 03:29:23.679475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:23.679435 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:23.984113 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:23.984032 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:29:23.984318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:23.984297 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" podUID="cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 03:29:24.104152 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:24.104114 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 03:29:24.104968 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:24.104718 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" podUID="60771f19-56cc-40a1-a27c-b09f23ac9903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 03:29:24.643703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:24.635336 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:24.985437 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:24.985384 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:29:24.985811 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:24.985753 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" podUID="cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 03:29:25.103411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:25.103368 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 03:29:25.103609 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:25.103449 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" podUID="60771f19-56cc-40a1-a27c-b09f23ac9903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 03:29:25.479778 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:25.479724 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:25.983704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:25.983651 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:29:25.983861 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:25.983727 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" podUID="cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 03:29:26.103851 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:26.103810 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 03:29:26.104031 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:26.103870 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" podUID="60771f19-56cc-40a1-a27c-b09f23ac9903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 03:29:26.209642 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:26.209604 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:26.861696 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:26.861396 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj"] Oct 31 03:29:26.871244 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:26.871200 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj"] Oct 31 03:29:26.871431 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:26.871321 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:26.881669 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:26.881003 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d86a77\"/\"distinct-2-w0-d86a77-kserve-self-signed-certs\"" Oct 31 03:29:26.921071 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:26.921033 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:26.963078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:26.963027 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-home\") pod \"distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:26.963078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:26.963075 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-tmp-dir\") pod \"distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:26.963363 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:26.963206 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xccbf\" (UniqueName: \"kubernetes.io/projected/3f89ed71-a623-40af-a5b1-5f6333235c14-kube-api-access-xccbf\") pod \"distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:26.963363 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:26.963247 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-model-cache\") pod \"distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:26.963492 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:26.963392 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-dshm\") pod \"distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:26.963492 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:26.963420 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f89ed71-a623-40af-a5b1-5f6333235c14-tls-certs\") pod \"distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:26.988842 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:26.988808 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:26.989141 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:26.989105 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:26.989994 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:26.989975 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:29:27.055491 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.055441 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:27.070853 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.068783 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-dshm\") pod \"distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:27.070853 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.069915 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f89ed71-a623-40af-a5b1-5f6333235c14-tls-certs\") pod \"distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:27.070853 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.070055 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-home\") pod \"distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:27.070853 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.070095 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-tmp-dir\") pod \"distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:27.070853 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.070227 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xccbf\" (UniqueName: \"kubernetes.io/projected/3f89ed71-a623-40af-a5b1-5f6333235c14-kube-api-access-xccbf\") pod \"distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:27.070853 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.070305 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-model-cache\") pod \"distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:27.072359 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.072321 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-model-cache\") pod \"distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:27.074292 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.073816 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-home\") pod \"distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:27.075057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.075032 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-tmp-dir\") pod \"distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:27.075753 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.075728 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-dshm\") pod \"distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:27.077516 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.077492 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f89ed71-a623-40af-a5b1-5f6333235c14-tls-certs\") pod \"distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:27.092646 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.092611 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xccbf\" (UniqueName: \"kubernetes.io/projected/3f89ed71-a623-40af-a5b1-5f6333235c14-kube-api-access-xccbf\") pod \"distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:27.104210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.104177 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 31 03:29:27.104389 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.104248 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" podUID="60771f19-56cc-40a1-a27c-b09f23ac9903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 31 03:29:27.190226 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.190190 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:27.489907 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.489876 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj"] Oct 31 03:29:27.491408 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:29:27.491372 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f89ed71_a623_40af_a5b1_5f6333235c14.slice/crio-e89ecef5458145ff96ece614a5e3d3a2c8d1bb1471850d64b193b3ee7cb881fe WatchSource:0}: Error finding container e89ecef5458145ff96ece614a5e3d3a2c8d1bb1471850d64b193b3ee7cb881fe: Status 404 returned error can't find the container with id e89ecef5458145ff96ece614a5e3d3a2c8d1bb1471850d64b193b3ee7cb881fe Oct 31 03:29:27.830377 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.830280 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:27.941524 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.941478 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" event={"ID":"3f89ed71-a623-40af-a5b1-5f6333235c14","Type":"ContainerStarted","Data":"47406f4963632d15d7744157303fb408aa472a0a64c5fcf2cfdc75f66054f241"} Oct 31 03:29:27.942030 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.941534 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" event={"ID":"3f89ed71-a623-40af-a5b1-5f6333235c14","Type":"ContainerStarted","Data":"e89ecef5458145ff96ece614a5e3d3a2c8d1bb1471850d64b193b3ee7cb881fe"} Oct 31 03:29:27.964089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:27.964027 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podStartSLOduration=1.964007163 podStartE2EDuration="1.964007163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:27.958195211 +0000 UTC m=+802.492890184" watchObservedRunningTime="2026-10-31 03:29:27.964007163 +0000 UTC m=+802.498702119" Oct 31 03:29:28.194805 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:28.194763 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:28.194999 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:28.194897 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:28.195607 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:28.195582 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:28.196584 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:28.196562 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:29:28.600406 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:28.599390 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:29.286635 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:29.285712 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:30.107357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:30.107324 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:30.947439 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:30.947403 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:31.698111 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:31.698067 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:32.397443 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:32.397401 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:33.111200 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.111167 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:33.678576 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.678536 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz"] Oct 31 03:29:33.683369 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.683336 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.702861 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.702829 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz"] Oct 31 03:29:33.769387 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.769341 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.769387 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.769389 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.769670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.769491 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjmrj\" (UniqueName: \"kubernetes.io/projected/77dfaec5-10f9-484c-ad1e-19980c14a3ea-kube-api-access-sjmrj\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.769670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.769559 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.769670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.769611 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.769910 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.769876 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.769990 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.769960 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.770064 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.770039 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.770225 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.770189 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.822461 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.822391 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:33.871211 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.871177 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.871423 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.871222 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.871423 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.871265 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.871423 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.871302 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.871423 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.871335 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.871634 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.871444 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.871634 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.871514 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjmrj\" (UniqueName: \"kubernetes.io/projected/77dfaec5-10f9-484c-ad1e-19980c14a3ea-kube-api-access-sjmrj\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.871634 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.871556 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.871634 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.871613 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.871877 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.871720 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.871935 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.871863 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.872109 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.872088 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.872178 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.872116 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.872350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.872333 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.874007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.873982 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.874207 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.874190 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.885307 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.885269 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.885307 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.885290 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjmrj\" (UniqueName: \"kubernetes.io/projected/77dfaec5-10f9-484c-ad1e-19980c14a3ea-kube-api-access-sjmrj\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:33.897409 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.897376 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d"] Oct 31 03:29:33.903351 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.903327 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:33.912826 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.912796 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f20dc4-openshift-default-dockercfg-zn4w2\"" Oct 31 03:29:33.916866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.916837 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d"] Oct 31 03:29:33.999104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:33.999012 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:34.074485 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.074442 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.074754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.074491 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.074754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.074522 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7qnp\" (UniqueName: \"kubernetes.io/projected/092826cc-3459-4bd9-b0f8-272ca8e44bf9-kube-api-access-t7qnp\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.074754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.074656 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.074754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.074733 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.075078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.074791 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.075078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.074874 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.075078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.074900 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.075078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.074969 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.115763 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.115030 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22"] Oct 31 03:29:34.120598 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.120572 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.126946 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.126892 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22"] Oct 31 03:29:34.175895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.175857 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.176062 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.175959 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.176062 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.176011 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.176192 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.176072 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.176192 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.176092 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.176192 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.176164 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.176340 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.176246 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.176340 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.176273 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.176340 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.176297 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7qnp\" (UniqueName: \"kubernetes.io/projected/092826cc-3459-4bd9-b0f8-272ca8e44bf9-kube-api-access-t7qnp\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.176542 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.176351 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.176858 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.176834 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.177132 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.177106 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.177132 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.177122 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.177252 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.177180 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.178236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.178219 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.178936 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.178920 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.184012 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.183991 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.184225 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.184203 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7qnp\" (UniqueName: \"kubernetes.io/projected/092826cc-3459-4bd9-b0f8-272ca8e44bf9-kube-api-access-t7qnp\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.220410 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.220378 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:34.277433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.277345 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.277599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.277441 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a305cab-5370-4be7-a94b-e2c8bb849726-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.277599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.277569 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.277711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.277632 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.277711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.277697 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.277891 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.277725 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgrtc\" (UniqueName: \"kubernetes.io/projected/0a305cab-5370-4be7-a94b-e2c8bb849726-kube-api-access-pgrtc\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.277891 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.277790 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.277891 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.277852 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.278103 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.277976 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.299979 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.299404 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz"] Oct 31 03:29:34.307971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.307389 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:34.307971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.307494 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:34.307971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.307543 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:34.378959 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.378920 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a305cab-5370-4be7-a94b-e2c8bb849726-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.379119 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.378983 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.379119 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.379011 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.379119 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.379041 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.379119 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.379058 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgrtc\" (UniqueName: \"kubernetes.io/projected/0a305cab-5370-4be7-a94b-e2c8bb849726-kube-api-access-pgrtc\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.379329 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.379129 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.379329 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.379170 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.379329 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.379266 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.379578 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.379353 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.379667 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.379646 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.379855 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.379829 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.379980 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.379935 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.380323 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.380300 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.380420 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.380335 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a305cab-5370-4be7-a94b-e2c8bb849726-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.382882 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.382861 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.383094 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.383075 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.386981 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.386959 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgrtc\" (UniqueName: \"kubernetes.io/projected/0a305cab-5370-4be7-a94b-e2c8bb849726-kube-api-access-pgrtc\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.387360 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.387340 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.441930 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.441862 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:34.517920 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.517620 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d"] Oct 31 03:29:34.533652 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:29:34.532818 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092826cc_3459_4bd9_b0f8_272ca8e44bf9.slice/crio-bc4b491932f97b2c77cb209237c9fe99dd96aa83a9f820af0eaf271ee0a3a4ea WatchSource:0}: Error finding container bc4b491932f97b2c77cb209237c9fe99dd96aa83a9f820af0eaf271ee0a3a4ea: Status 404 returned error can't find the container with id bc4b491932f97b2c77cb209237c9fe99dd96aa83a9f820af0eaf271ee0a3a4ea Oct 31 03:29:34.536969 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.535720 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:34.536969 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.535815 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:34.536969 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.535862 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:34.714664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.709533 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs"] Oct 31 03:29:34.742785 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.727026 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:34.742785 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.728346 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs"] Oct 31 03:29:34.742785 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.735134 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:34.772105 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.772074 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22"] Oct 31 03:29:34.781566 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:29:34.781482 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a305cab_5370_4be7_a94b_e2c8bb849726.slice/crio-6db6698983cca634bfc1fa1eccb9b6f6b7c9fa60a092203386d35a82302abed8 WatchSource:0}: Error finding container 6db6698983cca634bfc1fa1eccb9b6f6b7c9fa60a092203386d35a82302abed8: Status 404 returned error can't find the container with id 6db6698983cca634bfc1fa1eccb9b6f6b7c9fa60a092203386d35a82302abed8 Oct 31 03:29:34.798194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.797815 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:34.798194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.797918 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:34.798194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.797960 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:34.806221 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.806192 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv"] Oct 31 03:29:34.814914 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.814889 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:34.834201 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.834146 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv"] Oct 31 03:29:34.894884 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.894809 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z7hk\" (UniqueName: \"kubernetes.io/projected/de21da55-7c0d-4356-9fbb-d077817a9917-kube-api-access-7z7hk\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:34.895107 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.895080 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de21da55-7c0d-4356-9fbb-d077817a9917-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:34.895239 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.895221 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:34.895299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.895276 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de21da55-7c0d-4356-9fbb-d077817a9917-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:34.895353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.895302 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de21da55-7c0d-4356-9fbb-d077817a9917-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:34.895454 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.895436 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:34.895514 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.895490 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:34.895655 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.895639 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:34.895725 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:34.895710 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:35.001284 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.001238 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" event={"ID":"77dfaec5-10f9-484c-ad1e-19980c14a3ea","Type":"ContainerStarted","Data":"b3fca90efe6ceaf5b458e769f9a619be804526b5a65cc08f816a29381af91ea2"} Oct 31 03:29:35.001545 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.001521 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" event={"ID":"77dfaec5-10f9-484c-ad1e-19980c14a3ea","Type":"ContainerStarted","Data":"c074339ec2f4e3d02772718a3a6921abe88675e28ca1843ceb27cf9b8b3f53cb"} Oct 31 03:29:35.008247 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.007987 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.008805 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.008777 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" event={"ID":"092826cc-3459-4bd9-b0f8-272ca8e44bf9","Type":"ContainerStarted","Data":"8037bc405e75074c52647dc92cd935eb862fae882a28ccf157fa249b74b223ea"} Oct 31 03:29:35.009377 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.009353 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" event={"ID":"092826cc-3459-4bd9-b0f8-272ca8e44bf9","Type":"ContainerStarted","Data":"bc4b491932f97b2c77cb209237c9fe99dd96aa83a9f820af0eaf271ee0a3a4ea"} Oct 31 03:29:35.015834 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.015804 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" event={"ID":"0a305cab-5370-4be7-a94b-e2c8bb849726","Type":"ContainerStarted","Data":"81368f81741a3f5af04196f3bd7fd1586f9c6968340c7a4cafbf583f51637a62"} Oct 31 03:29:35.015937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.015860 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" event={"ID":"0a305cab-5370-4be7-a94b-e2c8bb849726","Type":"ContainerStarted","Data":"6db6698983cca634bfc1fa1eccb9b6f6b7c9fa60a092203386d35a82302abed8"} Oct 31 03:29:35.018739 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.016594 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:35.018739 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.016669 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.018739 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.018512 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de21da55-7c0d-4356-9fbb-d077817a9917-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:35.018739 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.018574 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.018739 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.018639 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de21da55-7c0d-4356-9fbb-d077817a9917-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:35.018739 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.018729 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:35.019114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.018793 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:35.019114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.018868 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.019114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.018906 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.019114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.018947 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.019114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.018998 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:35.019114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.019050 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.019442 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.019116 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z7hk\" (UniqueName: \"kubernetes.io/projected/de21da55-7c0d-4356-9fbb-d077817a9917-kube-api-access-7z7hk\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:35.020047 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.019591 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:35.021869 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.019619 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:35.021869 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.021097 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de21da55-7c0d-4356-9fbb-d077817a9917-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:35.021977 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.021363 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:35.022114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.022084 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:35.022778 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.022754 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.022927 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.022907 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v9h6\" (UniqueName: \"kubernetes.io/projected/5aa4a40b-944a-4113-9e1e-50c844f77cb4-kube-api-access-9v9h6\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.023025 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.023008 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de21da55-7c0d-4356-9fbb-d077817a9917-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:35.023101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.023046 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:35.026492 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.026445 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:35.027227 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.027204 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de21da55-7c0d-4356-9fbb-d077817a9917-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:35.061722 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.061577 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z7hk\" (UniqueName: \"kubernetes.io/projected/de21da55-7c0d-4356-9fbb-d077817a9917-kube-api-access-7z7hk\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:35.064264 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.064201 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de21da55-7c0d-4356-9fbb-d077817a9917-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:35.065313 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.065101 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:35.074708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.074027 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" podStartSLOduration=1.074013392 podStartE2EDuration="1.074013392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:35.073363249 +0000 UTC m=+809.608058208" watchObservedRunningTime="2026-10-31 03:29:35.074013392 +0000 UTC m=+809.608708347" Oct 31 03:29:35.081256 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.080840 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" podStartSLOduration=2.080819937 podStartE2EDuration="2.080819937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:35.053939757 +0000 UTC m=+809.588634712" watchObservedRunningTime="2026-10-31 03:29:35.080819937 +0000 UTC m=+809.615514894" Oct 31 03:29:35.108591 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.108406 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" podStartSLOduration=2.108384207 podStartE2EDuration="2.108384207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:35.100056815 +0000 UTC m=+809.634751771" watchObservedRunningTime="2026-10-31 03:29:35.108384207 +0000 UTC m=+809.643079162" Oct 31 03:29:35.143176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.126342 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.143176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.126393 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.143176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.126491 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.143176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.126585 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.143176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.126609 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.143176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.126637 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.143176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.126704 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.143176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.126807 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.143176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.126843 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v9h6\" (UniqueName: \"kubernetes.io/projected/5aa4a40b-944a-4113-9e1e-50c844f77cb4-kube-api-access-9v9h6\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.143176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.126989 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.143176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.127859 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.143176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.129451 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.143176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.129626 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.143176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.129880 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.143176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.130130 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.166053 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.166020 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.167795 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.167749 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.167918 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.167896 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v9h6\" (UniqueName: \"kubernetes.io/projected/5aa4a40b-944a-4113-9e1e-50c844f77cb4-kube-api-access-9v9h6\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.222616 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.222542 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:35.224457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.224158 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:29:35.224457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.224214 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" podUID="092826cc-3459-4bd9-b0f8-272ca8e44bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 03:29:35.469377 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.468192 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:35.473250 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.473006 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:35.473419 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.473344 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:29:35.473419 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.473391 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" podUID="0a305cab-5370-4be7-a94b-e2c8bb849726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 03:29:35.476609 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.476532 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:35.477263 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.477237 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs"] Oct 31 03:29:35.488145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.482641 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:35.488145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.482741 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:35.488145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:35.482790 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:36.004241 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:36.001301 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:36.028487 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:36.025615 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:29:36.028487 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:36.025703 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" podUID="77dfaec5-10f9-484c-ad1e-19980c14a3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:29:36.028487 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:36.026653 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:36.324532 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:36.321197 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:29:36.324532 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:36.321264 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" podUID="092826cc-3459-4bd9-b0f8-272ca8e44bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 03:29:36.458851 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:36.458813 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:29:36.474701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:36.471241 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" podUID="0a305cab-5370-4be7-a94b-e2c8bb849726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 03:29:36.578772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:36.552442 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" event={"ID":"de21da55-7c0d-4356-9fbb-d077817a9917","Type":"ContainerStarted","Data":"4cf7e95a6a4c43321ccd3dc5e7ae0f460bb4ed0defc31a2b929eea391cc414fe"} Oct 31 03:29:36.707194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:36.623708 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" podStartSLOduration=2.623670609 podStartE2EDuration="2.623670609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:36.600559199 +0000 UTC m=+811.135254158" watchObservedRunningTime="2026-10-31 03:29:36.623670609 +0000 UTC m=+811.158365569" Oct 31 03:29:36.760705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:36.750645 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv"] Oct 31 03:29:36.838585 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:36.838539 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:36.839954 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:36.839926 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:36.840584 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:36.840560 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:29:37.000946 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:37.000908 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:29:37.001211 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:37.001179 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" podUID="77dfaec5-10f9-484c-ad1e-19980c14a3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:29:37.082923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:37.082873 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:29:37.092857 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:37.092752 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" podUID="de21da55-7c0d-4356-9fbb-d077817a9917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:29:37.114101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:37.085742 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:37.259744 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:37.240430 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:29:37.259744 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:37.240486 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" podUID="092826cc-3459-4bd9-b0f8-272ca8e44bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 03:29:37.291903 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:37.291877 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:37.336275 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:37.336240 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:37.492637 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:37.492601 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:29:37.492952 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:37.492920 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" podUID="0a305cab-5370-4be7-a94b-e2c8bb849726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 03:29:37.508540 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:37.507807 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:37.745777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:37.728271 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:37.745777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:37.729290 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:37.783880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:37.783836 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" event={"ID":"5aa4a40b-944a-4113-9e1e-50c844f77cb4","Type":"ContainerStarted","Data":"be642f4ac7ec42e76fb26c1600da989765514131162edae6fdd3353d1cbce473"} Oct 31 03:29:37.801929 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:37.801889 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" event={"ID":"de21da55-7c0d-4356-9fbb-d077817a9917","Type":"ContainerStarted","Data":"09dfcaf647aa638d48c6af77846e8efd0eecf08730a021f0795436955fde104f"} Oct 31 03:29:38.009635 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:38.008100 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:29:38.009635 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:38.008165 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" podUID="77dfaec5-10f9-484c-ad1e-19980c14a3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:29:38.069308 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:38.068854 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:29:38.069308 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:38.068913 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" podUID="de21da55-7c0d-4356-9fbb-d077817a9917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:29:38.074611 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:38.073899 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:38.228100 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:38.220862 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:38.233073 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:38.232414 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:29:38.233073 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:38.232476 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" podUID="092826cc-3459-4bd9-b0f8-272ca8e44bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 03:29:38.302774 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:38.297053 5490 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:38.321808 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:38.319901 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:29:38.321808 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:38.320337 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:29:38.456940 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:38.456902 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:29:38.457227 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:38.457181 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" podUID="0a305cab-5370-4be7-a94b-e2c8bb849726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 03:29:38.512270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:38.512216 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:29:38.523839 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:38.518747 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:29:38.730954 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:38.730905 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:38.731724 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:38.731694 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:29:38.735709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:38.732371 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:38.735709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:38.732463 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:29:39.320729 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:39.316164 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:39.320729 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:39.316664 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:29:39.332699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:39.321793 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:29:39.332699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:39.321849 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" podUID="77dfaec5-10f9-484c-ad1e-19980c14a3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:29:39.332699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:39.322378 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:29:39.332699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:39.322415 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" podUID="de21da55-7c0d-4356-9fbb-d077817a9917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:29:39.425457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:39.425414 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:29:39.429985 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:39.429936 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:29:39.434704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:39.430509 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:29:39.434975 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:39.434959 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:29:39.536710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:39.515232 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:29:39.545823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:39.545771 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:29:39.582658 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:39.580853 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:29:39.582658 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:39.580937 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" podUID="092826cc-3459-4bd9-b0f8-272ca8e44bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 03:29:39.582658 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:39.581141 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:39.582658 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:39.545780 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" podUID="0a305cab-5370-4be7-a94b-e2c8bb849726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 03:29:39.612975 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:39.612945 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:29:39.711129 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:39.711038 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:39.888750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:39.819843 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:39.891812 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:39.891769 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:40.011822 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:40.011727 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:29:40.012092 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:40.012058 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" podUID="77dfaec5-10f9-484c-ad1e-19980c14a3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:29:40.192320 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:40.192281 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:29:40.192581 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:40.192548 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" podUID="de21da55-7c0d-4356-9fbb-d077817a9917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:29:40.326082 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:40.325986 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:29:40.352834 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:40.351143 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" podUID="092826cc-3459-4bd9-b0f8-272ca8e44bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 03:29:40.352834 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:40.351227 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:40.534700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:40.429924 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 31 03:29:40.534700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:40.430085 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:29:40.534700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:40.431856 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:40.534700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:40.528637 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" event={"ID":"5aa4a40b-944a-4113-9e1e-50c844f77cb4","Type":"ContainerStarted","Data":"ae26df749e7769606d444bf73ea3267a4b28f81ea175662e7c431999ab42e759"} Oct 31 03:29:40.554207 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:40.554161 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:29:40.554501 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:40.554464 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" podUID="0a305cab-5370-4be7-a94b-e2c8bb849726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 03:29:40.574269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:40.574216 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:40.597499 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:40.597374 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:29:40.748330 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:40.748292 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:41.004051 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:40.983889 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:29:41.037418 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:41.037372 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:41.037730 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:41.037706 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:29:41.083199 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:41.083161 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:29:41.083469 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:41.083443 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" podUID="de21da55-7c0d-4356-9fbb-d077817a9917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:29:41.115714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:41.107972 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:29:41.160690 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:41.160636 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" podUID="77dfaec5-10f9-484c-ad1e-19980c14a3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:29:41.187709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:41.185823 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" podUID="80180652-fb4f-4976-956d-25a47344d581" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:41.221062 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:41.221022 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:29:41.221312 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:41.221286 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" podUID="092826cc-3459-4bd9-b0f8-272ca8e44bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 03:29:41.364759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:41.362766 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:29:41.491704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:41.447619 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:29:41.532800 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:41.532755 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" podUID="0a305cab-5370-4be7-a94b-e2c8bb849726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 03:29:41.533447 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:41.505970 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:41.533620 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:41.472840 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:41.534068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:41.534030 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:41.534922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:41.534274 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:29:41.535134 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:41.535108 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" podUID="5aa4a40b-944a-4113-9e1e-50c844f77cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 03:29:41.591807 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:41.591759 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:41.598809 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:41.598760 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:41.599073 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:41.599051 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:29:42.009700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:42.007810 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:29:42.014698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:42.010934 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:29:42.014698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:42.010983 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" podUID="77dfaec5-10f9-484c-ad1e-19980c14a3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:29:42.046880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:42.046817 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:42.079754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:42.076618 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" podStartSLOduration=8.076597719 podStartE2EDuration="8.076597719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:41.158263623 +0000 UTC m=+815.692958579" watchObservedRunningTime="2026-10-31 03:29:42.076597719 +0000 UTC m=+816.611292675" Oct 31 03:29:42.088851 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:42.088808 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:29:42.091164 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:42.091127 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" podUID="de21da55-7c0d-4356-9fbb-d077817a9917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:29:42.224965 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:42.224927 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:29:42.225230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:42.225198 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" podUID="092826cc-3459-4bd9-b0f8-272ca8e44bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 03:29:42.456753 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:42.429509 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:42.486477 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:42.485841 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:29:42.486477 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:42.485916 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" podUID="0a305cab-5370-4be7-a94b-e2c8bb849726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 03:29:42.490702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:42.486865 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:29:42.490702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:42.486917 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" podUID="5aa4a40b-944a-4113-9e1e-50c844f77cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 03:29:42.539718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:42.534435 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:42.539718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:42.535222 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:42.608719 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:42.596997 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:42.620910 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:42.611183 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:29:42.640197 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:42.640117 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:43.010974 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:43.010926 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:43.037703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:43.014353 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:29:43.037703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:43.014415 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" podUID="77dfaec5-10f9-484c-ad1e-19980c14a3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:29:43.053810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:43.053766 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:43.081812 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:43.076106 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:29:43.081812 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:43.076183 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" podUID="de21da55-7c0d-4356-9fbb-d077817a9917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:29:43.224174 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:43.224136 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:29:43.224457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:43.224426 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" podUID="092826cc-3459-4bd9-b0f8-272ca8e44bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 03:29:43.430192 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:43.430123 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:43.443527 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:43.443491 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:29:43.443839 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:43.443809 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" podUID="0a305cab-5370-4be7-a94b-e2c8bb849726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 03:29:43.463199 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:43.463109 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:43.481191 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:43.481146 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:29:43.481488 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:43.481435 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" podUID="5aa4a40b-944a-4113-9e1e-50c844f77cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 03:29:43.537882 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:43.537820 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:29:43.539048 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:43.539018 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:43.604714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:43.601760 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:43.623712 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:43.621375 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:43.724889 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:43.724775 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:44.031277 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:44.031178 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:29:44.031522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:44.031491 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" podUID="77dfaec5-10f9-484c-ad1e-19980c14a3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:29:44.049739 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:44.034322 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:44.067769 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:44.063586 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:44.082725 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:44.071323 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:29:44.082725 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:44.071382 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" podUID="de21da55-7c0d-4356-9fbb-d077817a9917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:29:44.357759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:44.357294 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:44.410522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:44.410478 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:44.439894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:44.439839 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:29:44.467923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:44.467872 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:29:44.468558 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:44.468503 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" podUID="092826cc-3459-4bd9-b0f8-272ca8e44bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 03:29:44.495865 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:44.495828 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:29:44.520817 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:44.520771 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" podUID="5aa4a40b-944a-4113-9e1e-50c844f77cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 03:29:44.538707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:44.525326 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:29:44.538707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:44.525397 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" podUID="0a305cab-5370-4be7-a94b-e2c8bb849726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 03:29:44.564701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:44.546014 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:44.564701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:44.546286 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:29:44.606276 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:44.606234 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:44.654776 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:44.648443 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:45.076315 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:45.076211 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:45.077850 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:45.077813 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:29:45.122820 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:45.122782 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 03:29:45.123159 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:45.123124 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" podUID="77dfaec5-10f9-484c-ad1e-19980c14a3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 03:29:45.288370 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:45.288331 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:29:45.288634 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:45.288605 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" podUID="de21da55-7c0d-4356-9fbb-d077817a9917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:29:45.476702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:45.454081 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:45.539710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:45.537366 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:29:45.539710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:45.537433 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" podUID="0a305cab-5370-4be7-a94b-e2c8bb849726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 03:29:45.545518 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:45.543947 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:29:45.545518 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:45.544002 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" podUID="092826cc-3459-4bd9-b0f8-272ca8e44bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 03:29:45.545518 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:45.544538 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:29:45.545518 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:45.544585 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" podUID="5aa4a40b-944a-4113-9e1e-50c844f77cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 03:29:45.548390 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:45.546448 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:45.548390 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:45.546953 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:29:45.548390 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:45.547312 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:29:45.608823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:45.608768 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:45.693711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:45.685571 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:45.818852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:45.818752 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:29:46.112881 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:46.112772 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:46.148702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:46.127111 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:46.201456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:46.201416 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 03:29:46.201716 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:46.201667 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" podUID="de21da55-7c0d-4356-9fbb-d077817a9917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 03:29:46.221658 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:46.221624 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 31 03:29:46.221980 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:46.221935 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" podUID="092826cc-3459-4bd9-b0f8-272ca8e44bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 31 03:29:46.450975 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:46.450937 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 31 03:29:46.451329 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:46.451271 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" podUID="0a305cab-5370-4be7-a94b-e2c8bb849726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 31 03:29:46.478796 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:46.477332 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:29:46.512176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:46.512143 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:29:46.512460 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:46.512432 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" podUID="5aa4a40b-944a-4113-9e1e-50c844f77cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 03:29:46.659999 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:46.659864 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:29:46.868167 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:46.693816 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:46.881750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:46.720927 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:46.903021 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:46.902968 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:46.957028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:46.956990 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:46.970702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:46.958544 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:47.029718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.029660 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:29:47.110526 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.110496 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:29:47.130984 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.130854 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:47.133419 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.133393 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:29:47.586723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.586644 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:29:47.601752 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.601714 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:29:47.602223 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.602188 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" podUID="5aa4a40b-944a-4113-9e1e-50c844f77cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 03:29:47.717700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.717552 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:47.808194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.804980 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:47.808194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.805079 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:29:47.808194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.805122 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:29:47.808194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.805182 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:47.808194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.805228 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:29:47.808194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.808045 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:47.812118 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.812054 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:29:47.823295 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.813779 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:47.823295 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.814232 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:47.823295 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:29:47.822661 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:29:47.823295 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.822747 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:29:47.824647 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.824597 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:29:47.833197 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.833144 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:29:47.927031 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:47.926984 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:48.132105 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:48.132067 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:29:48.132390 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:48.132171 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" podUID="de21da55-7c0d-4356-9fbb-d077817a9917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:48.474735 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:48.474644 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 31 03:29:48.474947 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:48.474761 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" podUID="5aa4a40b-944a-4113-9e1e-50c844f77cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 31 03:29:48.915298 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:48.915233 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:49.016699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:49.015955 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:49.017971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:49.017225 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:49.022212 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:49.022186 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:29:49.479636 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:49.479605 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:49.479948 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:49.479929 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:49.480787 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:49.480767 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:29:49.616334 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:49.616294 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:50.395922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:50.395884 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:51.279104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:51.279067 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:51.989145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:51.989107 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:52.740511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:52.739278 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:53.407176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:53.407142 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:54.084620 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:54.083934 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:54.712326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:54.712280 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:55.384552 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:55.384503 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:55.800001 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:55.799965 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5"] Oct 31 03:29:55.805982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:55.805950 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:55.810406 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:55.810378 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5"] Oct 31 03:29:55.812881 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:55.812852 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-dda5cb\"/\"distinct-2-w4-dda5cb-kserve-self-signed-certs\"" Oct 31 03:29:55.879171 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:55.879136 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:55.924313 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:55.924279 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-dshm\") pod \"distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:55.924516 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:55.924324 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-tls-certs\") pod \"distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:55.924583 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:55.924518 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-model-cache\") pod \"distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:55.924583 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:55.924566 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-home\") pod \"distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:55.924735 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:55.924666 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk4wl\" (UniqueName: \"kubernetes.io/projected/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-kube-api-access-sk4wl\") pod \"distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:55.925052 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:55.924799 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-tmp-dir\") pod \"distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:56.026054 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:56.026011 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-model-cache\") pod \"distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:56.026255 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:56.026067 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-home\") pod \"distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:56.026255 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:56.026141 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk4wl\" (UniqueName: \"kubernetes.io/projected/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-kube-api-access-sk4wl\") pod \"distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:56.026255 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:56.026195 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-tmp-dir\") pod \"distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:56.026442 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:56.026373 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-dshm\") pod \"distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:56.026442 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:56.026410 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-tls-certs\") pod \"distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:56.026555 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:56.026482 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-model-cache\") pod \"distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:56.026652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:56.026627 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-home\") pod \"distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:56.026906 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:56.026885 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-tmp-dir\") pod \"distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:56.029350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:56.029326 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-dshm\") pod \"distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:56.029695 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:56.029656 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-tls-certs\") pod \"distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:56.041594 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:56.041563 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk4wl\" (UniqueName: \"kubernetes.io/projected/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-kube-api-access-sk4wl\") pod \"distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:56.122193 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:56.122099 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:56.177435 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:56.177398 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:56.419210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:56.419173 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5"] Oct 31 03:29:56.419427 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:29:56.419398 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ef5a4fe_922e_48f2_a273_bc8c3b3f0f66.slice/crio-fa1b029fa12002d7aa4c989d568457cb502f01904800f45dcbb64a41100bea0b WatchSource:0}: Error finding container fa1b029fa12002d7aa4c989d568457cb502f01904800f45dcbb64a41100bea0b: Status 404 returned error can't find the container with id fa1b029fa12002d7aa4c989d568457cb502f01904800f45dcbb64a41100bea0b Oct 31 03:29:56.849663 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:56.849571 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" event={"ID":"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66","Type":"ContainerStarted","Data":"f3bd1d126c4501d8caa00f35ce4915a45cce1f16610bd20d4ca66da3d7706806"} Oct 31 03:29:56.849663 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:56.849622 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" event={"ID":"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66","Type":"ContainerStarted","Data":"fa1b029fa12002d7aa4c989d568457cb502f01904800f45dcbb64a41100bea0b"} Oct 31 03:29:56.871568 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:56.871515 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podStartSLOduration=1.871499668 podStartE2EDuration="1.871499668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:56.870307139 +0000 UTC m=+831.405002097" watchObservedRunningTime="2026-10-31 03:29:56.871499668 +0000 UTC m=+831.406194623" Oct 31 03:29:56.906458 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:56.906418 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:57.123290 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:57.123203 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:29:57.678135 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:57.678094 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:29:59.373840 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:59.373779 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:59.538833 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:59.538784 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:59.540638 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:59.540318 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:29:59.577013 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:59.576966 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:59.649766 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:59.645053 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:29:59.911755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:59.900769 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:29:59.992834 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:29:59.992777 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:00.471951 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:00.471904 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:00.622088 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:00.496757 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:00.624463 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:00.539228 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:00.661700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:00.624624 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:30:00.662069 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:00.662042 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:30:00.663138 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:00.621733 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:30:00.805466 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:00.805378 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:30:00.857298 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:00.542760 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:00.895787 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:00.646753 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:00.895787 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:00.858867 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:30:00.895787 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:00.858935 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:30:00.895787 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:00.830749 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:01.012500 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:00.975647 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:01.012500 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:00.976343 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:30:01.055196 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:01.021585 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:30:01.055196 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:01.025224 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:30:01.055820 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:01.055738 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:01.615708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:01.613890 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:01.615708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:01.614060 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:01.651932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:01.645745 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:30:01.651932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:01.646117 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:30:01.810701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:01.809809 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:01.884768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:01.860797 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:01.884768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:01.861194 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:30:01.884768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:01.880633 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:01.987775 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:01.987718 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:30:02.039800 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:02.026782 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:02.039800 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:02.027396 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:30:02.371862 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:02.371820 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:30:02.372163 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:02.372129 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" podUID="5907e061-56b9-48e5-b9a8-9d33fe49e5a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:02.651798 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:02.646084 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:02.651798 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:02.647513 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:02.877703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:02.877421 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:30:02.920249 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:02.886294 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:30:02.920249 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:02.886586 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:30:02.920249 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:02.888615 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:03.043809 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:03.031836 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:03.043809 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:03.032778 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:30:03.043809 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:03.032828 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" podUID="77dfaec5-10f9-484c-ad1e-19980c14a3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:03.050754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:03.050708 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:03.512064 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:03.512015 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:03.515032 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:03.505707 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:03.664172 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:03.663539 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:03.664172 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:03.663891 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:30:03.732823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:03.725789 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:03.848495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:03.846529 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:30:03.884629 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:03.884593 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:30:03.886830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:03.886802 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:30:03.888293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:03.888267 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:30:03.889875 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:03.889112 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:30:03.908508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:03.891778 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:30:03.908508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:03.893065 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:30:03.925644 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:03.925613 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:30:04.382796 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.382754 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2"] Oct 31 03:30:04.402108 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.402064 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.414306 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.414272 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2"] Oct 31 03:30:04.517712 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.509528 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:04.541700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.538058 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.601641 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.538151 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.601884 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.601821 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.602746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.601988 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.602746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.602022 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.602746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.602116 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.602746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.602161 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs4zx\" (UniqueName: \"kubernetes.io/projected/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-kube-api-access-qs4zx\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.602746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.602229 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.602746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.602298 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.713705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.710744 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.713705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.710815 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.713705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.710911 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.713705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.710971 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs4zx\" (UniqueName: \"kubernetes.io/projected/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-kube-api-access-qs4zx\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.713705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.711036 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.713705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.711101 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.713705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.711198 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.728697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.720757 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.728697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.720846 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.736699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.729981 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.736699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.730891 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.736699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.731922 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.736699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.732081 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.742893 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.740950 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.762716 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.762286 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.770703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.766999 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.783699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.779514 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:04.828625 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:04.828586 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs4zx\" (UniqueName: \"kubernetes.io/projected/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-kube-api-access-qs4zx\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:05.048146 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:05.048044 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:05.306883 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:05.306775 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:05.370052 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:05.370021 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2"] Oct 31 03:30:05.371375 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:30:05.371349 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6d3eaf_534e_4de2_929e_e32cbb4d0ea9.slice/crio-16090692aeefcad852aa88f15bafa2d94b5ab7f39a2ee0a71112818e1a24d9b0 WatchSource:0}: Error finding container 16090692aeefcad852aa88f15bafa2d94b5ab7f39a2ee0a71112818e1a24d9b0: Status 404 returned error can't find the container with id 16090692aeefcad852aa88f15bafa2d94b5ab7f39a2ee0a71112818e1a24d9b0 Oct 31 03:30:05.374342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:05.374302 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:30:05.374421 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:05.374397 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:30:05.374468 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:05.374452 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:30:05.950304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:05.950260 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" event={"ID":"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9","Type":"ContainerStarted","Data":"8c00d36c8cc8ea0bb4deb8d74016fa1f87e9b2d0b188a91405ec1d54aa875bcd"} Oct 31 03:30:05.950801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:05.950360 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" event={"ID":"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9","Type":"ContainerStarted","Data":"16090692aeefcad852aa88f15bafa2d94b5ab7f39a2ee0a71112818e1a24d9b0"} Oct 31 03:30:05.972330 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:05.972262 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" podStartSLOduration=1.972241263 podStartE2EDuration="1.972241263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:05.969828844 +0000 UTC m=+840.504523801" watchObservedRunningTime="2026-10-31 03:30:05.972241263 +0000 UTC m=+840.506936219" Oct 31 03:30:06.048544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:06.048504 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:06.050109 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:06.050084 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:30:06.050301 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:06.050139 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" podUID="0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 03:30:06.123178 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:06.123145 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:30:06.137491 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:06.137440 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:30:06.200469 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:06.200369 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:06.972629 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:06.972589 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:30:06.980043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:06.980011 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:07.057981 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:07.057474 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:30:07.057981 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:07.057557 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" podUID="0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 03:30:08.018102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:08.018063 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:08.051053 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:08.050825 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:30:08.051053 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:08.050908 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" podUID="0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 03:30:08.916461 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:08.916423 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:09.056749 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:09.052263 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:30:09.056749 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:09.052339 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" podUID="0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 03:30:09.804133 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:09.804097 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:10.051553 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:10.051514 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:30:10.051907 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:10.051867 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" podUID="0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 03:30:10.715806 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:10.715756 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:11.052755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:11.052465 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:30:11.052755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:11.052540 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" podUID="0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 03:30:11.812479 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:11.812444 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:12.049077 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:12.049040 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 31 03:30:12.049275 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:12.049097 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" podUID="0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 31 03:30:12.583042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:12.583002 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:13.146254 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:13.146220 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:13.146736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:13.146713 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:13.147671 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:13.147644 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:30:13.397722 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:13.397583 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:14.112925 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:14.112652 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:15.004056 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:15.004007 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:15.665476 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:15.665426 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:15.803619 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:15.803532 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:16.507957 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:16.507920 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:17.129651 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:30:17.129608 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:30:17.368662 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:17.368297 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:17.980254 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:17.980220 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:18.594726 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:18.594624 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:19.218563 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.218422 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:19.224609 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.224568 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz"] Oct 31 03:30:19.253707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.253551 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz"] Oct 31 03:30:19.255477 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.253718 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.323950 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.323913 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.324124 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.323999 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4568c18-198b-4726-8005-35e3fdbae8d9-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.324124 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.324095 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.324207 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.324155 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.324304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.324274 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.324373 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.324358 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.324437 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.324422 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.324491 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.324475 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zczv\" (UniqueName: \"kubernetes.io/projected/d4568c18-198b-4726-8005-35e3fdbae8d9-kube-api-access-2zczv\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.324569 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.324548 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.426012 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.425975 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.426210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.426028 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.426210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.426059 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.426210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.426083 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.426210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.426115 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zczv\" (UniqueName: \"kubernetes.io/projected/d4568c18-198b-4726-8005-35e3fdbae8d9-kube-api-access-2zczv\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.426210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.426162 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.426481 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.426243 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.426481 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.426332 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4568c18-198b-4726-8005-35e3fdbae8d9-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.426481 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.426411 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.426481 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.426448 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.426661 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.426521 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.426661 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.426556 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.426941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.426919 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.427028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.427008 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4568c18-198b-4726-8005-35e3fdbae8d9-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.428480 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.428457 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.428810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.428793 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.434015 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.433989 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.434195 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.434176 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zczv\" (UniqueName: \"kubernetes.io/projected/d4568c18-198b-4726-8005-35e3fdbae8d9-kube-api-access-2zczv\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.525833 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.525654 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp"] Oct 31 03:30:19.531414 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.531386 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.545296 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.545267 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp"] Oct 31 03:30:19.563057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.563016 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2"] Oct 31 03:30:19.569157 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.569126 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.583238 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.583205 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2"] Oct 31 03:30:19.586074 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.586046 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:19.629098 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.629053 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.629302 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.629133 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.629302 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.629225 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.629302 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.629290 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.629483 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.629316 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vbt4\" (UniqueName: \"kubernetes.io/projected/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-kube-api-access-4vbt4\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.629483 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.629346 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.629483 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.629381 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.629483 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.629429 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.629483 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.629466 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.629782 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.629495 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.629782 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.629530 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.629782 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.629632 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.629782 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.629738 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.631115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.631072 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.631248 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.631157 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.631248 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.631189 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgjmd\" (UniqueName: \"kubernetes.io/projected/aca6c54e-0673-4a1d-9e32-d653489ec2cf-kube-api-access-mgjmd\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.631368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.631250 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.631368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.631316 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.732806 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.732761 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.733012 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.732845 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.733012 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.732892 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.733012 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.732917 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vbt4\" (UniqueName: \"kubernetes.io/projected/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-kube-api-access-4vbt4\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.733012 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.732947 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.733244 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.733004 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.733244 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.733075 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.733244 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.733114 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.733244 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.733146 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.733244 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.733183 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.733501 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.733284 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.733501 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.733319 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.733501 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.733336 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.733501 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.733362 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.733501 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.733432 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.733830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.733797 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.733988 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.733948 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgjmd\" (UniqueName: \"kubernetes.io/projected/aca6c54e-0673-4a1d-9e32-d653489ec2cf-kube-api-access-mgjmd\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.734142 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.734117 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.734263 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.734248 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.734357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.734344 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.735164 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.735137 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.735745 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.735719 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.735985 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.735962 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.736166 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.736139 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.736455 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.736433 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.736530 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.736454 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.736530 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.736464 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.737001 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.736975 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.737113 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.737095 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.737618 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.737595 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.737770 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.737656 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.738191 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.738170 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.742072 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.742045 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.748771 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.748744 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vbt4\" (UniqueName: \"kubernetes.io/projected/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-kube-api-access-4vbt4\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.748907 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.748756 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgjmd\" (UniqueName: \"kubernetes.io/projected/aca6c54e-0673-4a1d-9e32-d653489ec2cf-kube-api-access-mgjmd\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.749381 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.749260 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.852078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.851987 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:19.895843 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.895805 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:19.896413 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.896360 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz"] Oct 31 03:30:19.907876 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.907839 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb"] Oct 31 03:30:19.954240 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.954207 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb"] Oct 31 03:30:19.954459 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.954435 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:19.959777 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:30:19.959751 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4568c18_198b_4726_8005_35e3fdbae8d9.slice/crio-7ade8bb6cc5966415a4feba3509d2178cdfac20d7b96283ba938a031fb227318 WatchSource:0}: Error finding container 7ade8bb6cc5966415a4feba3509d2178cdfac20d7b96283ba938a031fb227318: Status 404 returned error can't find the container with id 7ade8bb6cc5966415a4feba3509d2178cdfac20d7b96283ba938a031fb227318 Oct 31 03:30:19.984519 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.984475 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:30:19.984665 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.984546 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:30:19.984665 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:19.984579 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:30:20.048356 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.048313 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.048603 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.048424 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.048603 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.048536 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.048603 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.048577 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.048823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.048609 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srxjn\" (UniqueName: \"kubernetes.io/projected/93fab1f3-4ba5-46fa-a9f6-001946891f5b-kube-api-access-srxjn\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.048823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.048651 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.048823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.048773 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.048823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.048805 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.049048 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.048918 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.088973 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.088151 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:20.117723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.117612 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" event={"ID":"d4568c18-198b-4726-8005-35e3fdbae8d9","Type":"ContainerStarted","Data":"015ef7b18acfc102e026aa5beba3a3a463b052da244b8d1d08d16fcced18b268"} Oct 31 03:30:20.118284 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.118256 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" event={"ID":"d4568c18-198b-4726-8005-35e3fdbae8d9","Type":"ContainerStarted","Data":"7ade8bb6cc5966415a4feba3509d2178cdfac20d7b96283ba938a031fb227318"} Oct 31 03:30:20.173330 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.160584 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.173330 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.161020 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.173330 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.161090 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.173330 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.161549 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.173330 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.161944 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.173330 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.162312 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.173330 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.162376 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srxjn\" (UniqueName: \"kubernetes.io/projected/93fab1f3-4ba5-46fa-a9f6-001946891f5b-kube-api-access-srxjn\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.173330 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.172426 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.173330 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.172552 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.173330 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.172715 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.173330 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.172760 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.173330 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.172980 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.177522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.175002 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.177522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.176149 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.180342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.180283 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.185885 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.185823 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" podStartSLOduration=1.185802482 podStartE2EDuration="1.185802482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:20.148269444 +0000 UTC m=+854.682964399" watchObservedRunningTime="2026-10-31 03:30:20.185802482 +0000 UTC m=+854.720497437" Oct 31 03:30:20.186827 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.186794 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.188909 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.188875 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.190481 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.190453 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srxjn\" (UniqueName: \"kubernetes.io/projected/93fab1f3-4ba5-46fa-a9f6-001946891f5b-kube-api-access-srxjn\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.225404 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.225374 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp"] Oct 31 03:30:20.227027 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:30:20.226997 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94f248b_c248_4eb1_ac7c_9d22f5cec9bf.slice/crio-d8292039bad2763264aeaab7c64d493ec90922a0c149d621d2f09a482ded66d0 WatchSource:0}: Error finding container d8292039bad2763264aeaab7c64d493ec90922a0c149d621d2f09a482ded66d0: Status 404 returned error can't find the container with id d8292039bad2763264aeaab7c64d493ec90922a0c149d621d2f09a482ded66d0 Oct 31 03:30:20.230560 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.230523 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:30:20.230626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.230607 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:30:20.230668 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.230651 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:30:20.290464 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.290439 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:20.300249 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.300211 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2"] Oct 31 03:30:20.305362 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:30:20.305334 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca6c54e_0673_4a1d_9e32_d653489ec2cf.slice/crio-3c115cd8f7f47bff5bf00cd1d562eacf41cdf5739584bf09e34bb2176886dd58 WatchSource:0}: Error finding container 3c115cd8f7f47bff5bf00cd1d562eacf41cdf5739584bf09e34bb2176886dd58: Status 404 returned error can't find the container with id 3c115cd8f7f47bff5bf00cd1d562eacf41cdf5739584bf09e34bb2176886dd58 Oct 31 03:30:20.309086 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.309050 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:30:20.309211 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.309142 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:30:20.309211 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.309196 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:30:20.586511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.586479 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:20.644793 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.637263 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb"] Oct 31 03:30:20.644793 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.642135 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:30:20.644793 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.642223 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:30:20.644793 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.642299 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:30:20.980960 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:20.978133 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:21.146698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:21.145823 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" event={"ID":"aca6c54e-0673-4a1d-9e32-d653489ec2cf","Type":"ContainerStarted","Data":"2cd048810fd7457188bb6b9d1083df1b24c2c28593dfe06992dd0137eb0f80d5"} Oct 31 03:30:21.146698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:21.145934 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" event={"ID":"aca6c54e-0673-4a1d-9e32-d653489ec2cf","Type":"ContainerStarted","Data":"3c115cd8f7f47bff5bf00cd1d562eacf41cdf5739584bf09e34bb2176886dd58"} Oct 31 03:30:21.166421 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:21.166378 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" event={"ID":"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf","Type":"ContainerStarted","Data":"602aee522b4370dd4d6334ca9f0a1807409b409dfd015cf3fef751036df1003f"} Oct 31 03:30:21.166582 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:21.166450 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" event={"ID":"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf","Type":"ContainerStarted","Data":"d8292039bad2763264aeaab7c64d493ec90922a0c149d621d2f09a482ded66d0"} Oct 31 03:30:21.168627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:21.168597 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" event={"ID":"93fab1f3-4ba5-46fa-a9f6-001946891f5b","Type":"ContainerStarted","Data":"80ceb5748003ba6aad148b3cbff454e941c905cce04047854876fbda303e7923"} Oct 31 03:30:21.168810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:21.168652 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" event={"ID":"93fab1f3-4ba5-46fa-a9f6-001946891f5b","Type":"ContainerStarted","Data":"e289c9d17dd8925758bb57e0357c640b3aec9eb77df73fb056a34a936965d464"} Oct 31 03:30:21.182514 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:21.182450 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" podStartSLOduration=2.182429638 podStartE2EDuration="2.182429638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:21.176223062 +0000 UTC m=+855.710918030" watchObservedRunningTime="2026-10-31 03:30:21.182429638 +0000 UTC m=+855.717124593" Oct 31 03:30:21.199193 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:21.199130 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" podStartSLOduration=2.199109235 podStartE2EDuration="2.199109235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:21.195081703 +0000 UTC m=+855.729776660" watchObservedRunningTime="2026-10-31 03:30:21.199109235 +0000 UTC m=+855.733804191" Oct 31 03:30:21.218307 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:21.218248 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" podStartSLOduration=2.21823241 podStartE2EDuration="2.21823241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:21.214119886 +0000 UTC m=+855.748814839" watchObservedRunningTime="2026-10-31 03:30:21.21823241 +0000 UTC m=+855.752927361" Oct 31 03:30:21.291456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:21.291305 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:21.292954 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:21.292924 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 03:30:21.293081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:21.292979 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" podUID="93fab1f3-4ba5-46fa-a9f6-001946891f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 03:30:21.614382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:21.613966 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:30:21.614382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:21.614028 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" podUID="d4568c18-198b-4726-8005-35e3fdbae8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:30:21.855706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:21.855037 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:21.855706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:21.855183 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:30:21.855706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:21.855229 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" podUID="b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:30:21.938750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:21.918693 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:21.938750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:21.918860 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:30:21.938750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:21.918907 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" podUID="aca6c54e-0673-4a1d-9e32-d653489ec2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:30:22.280762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:22.278328 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:22.292736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:22.292133 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 03:30:22.292736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:22.292201 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" podUID="93fab1f3-4ba5-46fa-a9f6-001946891f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 03:30:22.590419 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:22.590320 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:30:22.590710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:22.590660 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" podUID="d4568c18-198b-4726-8005-35e3fdbae8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:30:22.650700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:22.649778 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:30:22.650700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:22.650194 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:22.857201 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:22.857104 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:30:22.857487 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:22.857456 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" podUID="b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:30:22.885832 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:22.885775 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:22.886524 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:22.886081 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:22.887828 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:22.887799 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:22.888661 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:22.888386 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:22.898085 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:22.898052 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:30:22.898327 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:22.898300 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" podUID="aca6c54e-0673-4a1d-9e32-d653489ec2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:30:22.981870 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:22.981803 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:23.334702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.312942 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 03:30:23.335360 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.335314 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" podUID="93fab1f3-4ba5-46fa-a9f6-001946891f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 03:30:23.387713 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.386734 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:23.463704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.459922 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:23.588758 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.588520 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:30:23.588758 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.588581 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" podUID="d4568c18-198b-4726-8005-35e3fdbae8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:30:23.709862 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.709796 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:23.744708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.739603 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:23.744708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.743217 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Oct 31 03:30:24.048076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.916316 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:30:24.048076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.938825 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:30:24.048076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.940311 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:30:24.048076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.940592 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:30:24.048076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.942453 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:30:24.048076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.942507 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" podUID="b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:30:24.048076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.942824 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:30:24.048076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.942859 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" podUID="aca6c54e-0673-4a1d-9e32-d653489ec2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:30:24.048076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.966786 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:24.048076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.967226 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:30:24.048076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.997832 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:24.048076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:23.998617 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:30:24.073219 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:24.062192 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:30:24.318893 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:24.318795 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 03:30:24.319151 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:24.319121 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" podUID="93fab1f3-4ba5-46fa-a9f6-001946891f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 03:30:24.707764 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:24.670553 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:30:24.707764 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:24.676908 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:30:24.707764 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:24.685235 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" podUID="d4568c18-198b-4726-8005-35e3fdbae8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:30:24.793116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:24.760885 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:30:24.793116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:24.762278 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:30:24.800776 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:24.800666 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:24.812308 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:24.811814 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:30:24.904595 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:24.904555 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:30:24.904877 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:24.904840 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" podUID="b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:30:24.948701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:24.941962 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:24.948701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:24.942756 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:30:24.964760 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:24.950402 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:30:24.965046 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:24.965002 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" podUID="aca6c54e-0673-4a1d-9e32-d653489ec2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:30:24.967310 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:24.967274 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:24.992856 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:24.992816 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:25.097702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:25.066976 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:25.097702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:25.078891 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:30:25.298825 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:25.295238 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 03:30:25.298825 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:25.295305 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" podUID="93fab1f3-4ba5-46fa-a9f6-001946891f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 03:30:25.686740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:25.655879 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:30:25.686740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:25.655975 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" podUID="d4568c18-198b-4726-8005-35e3fdbae8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:30:25.771957 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:25.771909 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:25.803981 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:25.783874 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:30:25.832927 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:25.832875 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:30:26.026015 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:25.866729 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:30:26.026506 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:26.026459 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" podUID="b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:30:26.026716 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:25.950251 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:26.051700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:25.913791 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:26.052015 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:25.991095 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:26.052319 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:25.950292 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:26.052442 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:25.991045 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:30:26.052515 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:26.052470 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" podUID="aca6c54e-0673-4a1d-9e32-d653489ec2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:30:26.077715 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:26.076811 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:26.361635 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:26.236028 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:26.361635 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:26.341833 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 03:30:26.361635 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:26.341911 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" podUID="93fab1f3-4ba5-46fa-a9f6-001946891f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 03:30:26.633759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:26.629590 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:30:26.673857 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:26.655593 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" podUID="d4568c18-198b-4726-8005-35e3fdbae8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:30:26.775843 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:30:26.774659 5490 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.175s" Oct 31 03:30:26.775843 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:26.774804 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:30:26.775843 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:26.774915 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:30:26.840711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:26.839956 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:30:26.882089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:26.851977 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:27.003741 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:26.970499 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:27.054970 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:27.054905 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:27.078004 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:27.077944 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:27.102593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:27.079816 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:30:27.102593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:27.080557 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:30:27.102593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:27.083141 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:27.102593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:27.098842 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:27.131703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:27.102966 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:30:27.140818 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:27.140771 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" podUID="b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:30:27.280765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:27.128418 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:27.346380 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:27.128358 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:30:27.426770 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:27.133781 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:30:27.513725 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:27.489351 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" podUID="aca6c54e-0673-4a1d-9e32-d653489ec2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:30:27.587835 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:27.545829 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 03:30:27.613599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:27.613545 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" podUID="93fab1f3-4ba5-46fa-a9f6-001946891f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 03:30:27.638702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:27.632026 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:30:27.705710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:27.635851 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" podUID="d4568c18-198b-4726-8005-35e3fdbae8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:30:27.847023 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:27.846919 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:27.888921 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:27.888845 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:28.116525 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.116423 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:30:28.174737 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.171126 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:30:28.188988 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.184241 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:30:28.324117 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.324047 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" podUID="775a8940-cb22-4774-a43f-c6e7c5364354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:28.324794 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.324753 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:28.325128 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.325099 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:28.426763 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.386956 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:30:28.426763 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.396794 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 03:30:28.426763 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.396846 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" podUID="93fab1f3-4ba5-46fa-a9f6-001946891f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 03:30:28.451702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.444301 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:30:28.526731 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.449771 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:30:28.554557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.454111 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:28.554557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.464635 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" podUID="9e7f6e11-0382-4897-88e3-1d4283f5bed3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.49:50052\" within 1s: context deadline exceeded" Oct 31 03:30:28.554557 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:30:28.465415 5490 logging.go:55] [core] [Channel #531 SubChannel #534]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.49:50052", ServerName: "10.133.0.49:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.49:50052: operation was canceled" Oct 31 03:30:28.554557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.473662 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:30:28.554557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.545848 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" podUID="aca6c54e-0673-4a1d-9e32-d653489ec2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:30:28.554557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.488420 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:30:28.554557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.545988 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:30:28.554557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.546033 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:30:28.554557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.473729 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:30:28.554557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.546221 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" podUID="b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:30:28.566292 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.555307 5490 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-5kbd8 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.11:9443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:30:28.566579 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.566539 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" podUID="8bae2714-9c1b-44db-a6ef-b0b008cf1fa2" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.133.0.11:9443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:28.587305 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.587269 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:30:28.587730 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.587696 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" podUID="d4568c18-198b-4726-8005-35e3fdbae8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:30:28.627418 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.624867 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:30:28.721975 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.721235 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:30:28.808150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.807130 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:30:28.889436 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.889383 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:28.902361 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.894980 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:30:28.902361 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.896777 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 31 03:30:28.902361 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.896825 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" podUID="b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 31 03:30:28.935623 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.935583 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:30:28.936225 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.936191 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" podUID="aca6c54e-0673-4a1d-9e32-d653489ec2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:30:29.001990 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:28.995978 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:30:29.005724 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:29.002543 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:30:29.015280 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:29.015231 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:29.160367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:29.160320 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:30:29.307754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:29.307100 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 03:30:29.307754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:29.307160 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" podUID="93fab1f3-4ba5-46fa-a9f6-001946891f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 03:30:29.679834 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:29.679792 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 03:30:29.680319 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:29.680188 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" podUID="d4568c18-198b-4726-8005-35e3fdbae8d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 03:30:29.897120 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:29.897082 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 03:30:29.897880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:29.897839 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" podUID="aca6c54e-0673-4a1d-9e32-d653489ec2cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 03:30:29.989875 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:29.989599 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:30.304927 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:30.304834 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 31 03:30:30.315799 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:30.315752 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" podUID="93fab1f3-4ba5-46fa-a9f6-001946891f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 31 03:30:30.400168 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:30.399075 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:30.404491 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:30.404463 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:30.414373 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:30.414304 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:30:30.657790 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:30.657747 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:30.658973 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:30.658952 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:30.660612 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:30.660577 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:30:30.904319 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:30.904283 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:30.905647 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:30.905343 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:30.906410 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:30.906388 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:30:31.045961 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:31.045867 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:31.381406 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:31.381313 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:31.381973 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:31.381948 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:31.382850 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:31.382831 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:30:32.038389 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:32.038345 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:33.742168 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:33.740937 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Oct 31 03:30:33.742168 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:33.741311 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:33.788313 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:33.788255 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:33.851473 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:33.851422 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:30:33.889022 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:33.888964 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:33.901714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:33.901431 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:30:34.041764 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:34.041000 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:34.065710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:34.064939 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:34.105699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:34.089559 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:34.299952 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:34.299812 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:34.350905 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:34.350840 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:34.745702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:34.744784 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:34.745702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:34.745274 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:30:34.941272 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:34.941214 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:34.975886 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:34.942357 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:30:34.975886 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:34.943743 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:34.975886 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:34.944423 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:30:35.012773 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:34.978627 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:30:35.013766 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:35.003810 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:35.126236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:35.016108 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:30:35.126236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:35.016184 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:30:35.126236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:35.069324 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:35.126236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:35.080811 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:30:35.126236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:35.095798 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:30:35.126236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:35.095869 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:30:35.126236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:35.103133 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:35.126236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:35.115072 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:30:35.255777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:35.245895 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:35.263571 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:35.263471 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:35.372724 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:35.364661 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:35.377710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:35.374373 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:30:35.737188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:35.627107 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:30:35.740700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:35.737467 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" podUID="5aa4a40b-944a-4113-9e1e-50c844f77cb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:35.759242 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:35.759193 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:35.977760 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:35.962847 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:36.008790 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:35.962882 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:30:36.032923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:36.031797 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:36.068703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:36.066110 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:36.069755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:36.069715 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:30:36.081822 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:36.081771 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:30:36.133244 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:36.133196 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:36.365893 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:36.365776 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:36.894703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:36.893007 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:36.994703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:36.991344 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:37.041910 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:37.026397 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:30:37.042357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:37.026551 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:37.049709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:37.043473 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:30:37.088703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:37.085070 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:30:37.108551 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:37.108505 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:30:37.109096 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:37.109061 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:37.142735 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:37.136787 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:37.199827 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:37.199776 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:37.640819 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:37.609021 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:38.103717 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:38.101902 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:30:38.176923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:38.176866 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:38.177636 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:38.177603 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:30:38.178169 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:38.178143 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:30:38.179314 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:38.179285 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:38.257317 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:38.206445 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:30:38.337710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:38.208222 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:30:38.372762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:38.208365 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:38.372762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:38.210492 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:38.480811 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:38.450748 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:30:38.680075 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:38.680032 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:30:39.138772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:39.116789 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:39.206762 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:30:39.172986 5490 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.561s" Oct 31 03:30:39.224059 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:39.224008 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:30:39.225923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:39.225888 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:30:39.231824 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:39.231788 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:30:39.299883 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:39.287366 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:39.514702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:39.509904 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:30:39.540703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:39.539770 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:30:39.544815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:39.544761 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" podUID="80180652-fb4f-4976-956d-25a47344d581" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:39.989716 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:39.983661 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:30:40.047384 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:40.047334 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:30:40.284751 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:40.191846 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:40.346486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:40.224755 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:30:40.346486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:40.224811 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:40.346486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:40.227424 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:40.346486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:40.241433 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:40.380871 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:40.380826 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:30:40.574578 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:40.574477 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:30:40.582153 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:40.582111 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 31 03:30:40.592587 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:40.592544 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:30:40.595105 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:40.595084 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:30:40.606739 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:40.592879 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:30:40.609493 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:40.609466 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:30:40.609712 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:40.609699 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:30:40.609831 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:40.609821 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:30:40.609946 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:40.609936 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:30:40.610091 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:40.610078 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:30:40.638883 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:40.638848 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:30:40.641707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:40.641665 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:30:41.286800 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:41.286758 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:42.003273 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:42.002982 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:42.877995 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:42.877949 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:43.706565 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:43.706519 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:44.445081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:44.445037 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:45.165921 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:45.165882 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:45.891629 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:45.891590 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:46.591793 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:46.591752 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:47.128838 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:30:47.128792 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:30:47.296491 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:47.296431 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:48.103615 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:48.103553 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:48.875866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:48.875832 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:49.577784 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:49.577748 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:50.386946 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:50.386909 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:50.987142 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:50.987101 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:51.712714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:51.710013 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:52.418987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:52.418934 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:53.290079 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:53.290043 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:54.068056 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:54.068020 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:54.844708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:54.844602 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:55.609825 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:55.609775 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:56.203542 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:56.203500 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:56.954554 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:56.954513 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:57.613146 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:57.613099 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:58.339830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:58.339785 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:59.008799 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:59.008750 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:30:59.611919 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:30:59.611877 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:00.946703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:00.937780 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:01.028777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:01.023197 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:01.028777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:01.024361 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:01.050702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:01.042163 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:31:01.182703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:01.181779 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:01.205763 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:01.189756 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:31:01.380704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:01.362786 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:01.598772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:01.597954 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Oct 31 03:31:01.621701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:01.614887 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:01.686894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:01.685796 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:31:02.111699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:02.109062 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:31:02.111699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:02.110593 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:31:02.137765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:02.123467 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:02.137765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:02.123795 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:31:02.137765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:02.124432 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:31:02.137765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:02.124491 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:31:02.317701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:02.154515 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:02.317701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:02.238100 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:02.317701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:02.291570 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:31:02.317701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:02.291987 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:31:02.317701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:02.292786 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:31:02.317701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:02.292855 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:31:02.365763 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:02.365423 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:02.365954 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:02.365915 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:31:02.546703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:02.520783 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:31:02.546703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:02.520857 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" podUID="93fab1f3-4ba5-46fa-a9f6-001946891f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:02.620897 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:02.620794 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:02.647238 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:02.625402 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:31:02.687794 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:02.687752 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:31:02.696750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:02.696712 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:31:03.126698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:03.125048 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:03.190732 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:03.153351 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:03.190732 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:03.159977 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:31:03.190732 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:03.175775 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:03.190732 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:03.176132 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:31:03.190732 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:03.185798 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:31:03.190732 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:03.185866 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" podUID="77dfaec5-10f9-484c-ad1e-19980c14a3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:03.297719 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:03.293381 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:03.297719 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:03.294597 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:03.688813 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:03.688760 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:03.691472 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:03.691430 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:03.755492 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:03.755441 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Oct 31 03:31:03.801703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:03.799781 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:31:03.801703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:03.800319 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Oct 31 03:31:04.017257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:04.017154 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:31:04.017526 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:04.017493 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" podUID="092826cc-3459-4bd9-b0f8-272ca8e44bf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:04.371201 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:04.371089 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:31:04.401845 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:04.387129 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:04.430617 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:04.430552 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:04.732146 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:04.732100 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:04.732552 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:04.732523 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:31:04.874902 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:04.857711 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:04.943788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:04.935945 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:31:04.981469 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:04.981427 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:05.005445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:04.988450 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:31:05.005445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:04.988750 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:05.005445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:04.968670 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:31:05.005445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:04.989935 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:31:05.005445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:04.989974 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:31:05.005445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:04.990091 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p"] Oct 31 03:31:05.045923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.045890 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:31:05.046163 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.046145 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p"] Oct 31 03:31:05.046260 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.046244 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.046381 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.046285 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv"] Oct 31 03:31:05.061936 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.061909 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:31:05.062124 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.062112 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv"] Oct 31 03:31:05.062298 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.062278 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:31:05.062412 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.062402 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:31:05.062522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.062512 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:31:05.062641 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.062629 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:31:05.063028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.062049 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.194587 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.194546 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5rrn\" (UniqueName: \"kubernetes.io/projected/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-kube-api-access-h5rrn\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.194857 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.194814 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.194947 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.194891 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.195046 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.195006 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.195318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.195295 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.195588 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.195572 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.195625 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.195596 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.195663 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.195640 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfffj\" (UniqueName: \"kubernetes.io/projected/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-kube-api-access-vfffj\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.195762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.195661 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.196228 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.196196 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.196362 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.196254 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.196362 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.196300 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.196362 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.196335 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.196637 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.196418 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.196924 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.196898 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.197313 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.197280 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.197421 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.197360 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.197421 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.197409 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.298851 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.298737 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.299370 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.299321 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.299480 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.299405 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.299480 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.299462 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.299597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.299489 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.299597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.299522 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.299597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.299573 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.299783 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.299660 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.299783 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.299719 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.299783 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.299757 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.299945 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.299897 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5rrn\" (UniqueName: \"kubernetes.io/projected/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-kube-api-access-h5rrn\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.300003 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.299984 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.300113 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.300030 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.300113 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.300072 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.300224 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.300113 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.300224 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.300195 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.300224 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.300220 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.300385 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.300260 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfffj\" (UniqueName: \"kubernetes.io/projected/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-kube-api-access-vfffj\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.300385 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.300293 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.301261 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.301233 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.301495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.301474 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.302884 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.302315 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.303923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.302837 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.303923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.302848 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.303923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.303110 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.303923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.303474 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.303923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.303740 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.303923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.303769 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.304837 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.304811 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.306474 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.306451 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.306661 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.306638 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.307490 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.307464 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.311471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.311444 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.311737 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.311710 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfffj\" (UniqueName: \"kubernetes.io/projected/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-kube-api-access-vfffj\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.311911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.311886 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.311997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.311972 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5rrn\" (UniqueName: \"kubernetes.io/projected/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-kube-api-access-h5rrn\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.376061 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.376025 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:05.384999 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.384974 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:05.644762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.642375 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:05.694849 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.694818 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p"] Oct 31 03:31:05.707835 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.705930 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:31:05.707835 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.706033 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:31:05.707835 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.706082 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:31:05.756864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.756009 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv"] Oct 31 03:31:05.763776 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:31:05.763743 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c13f49_3afa_4080_bcaf_e9fcba5f5fc7.slice/crio-4bafe6780a261405095d6783541ec7c61c313ecf175969ed1c57576bca444527 WatchSource:0}: Error finding container 4bafe6780a261405095d6783541ec7c61c313ecf175969ed1c57576bca444527: Status 404 returned error can't find the container with id 4bafe6780a261405095d6783541ec7c61c313ecf175969ed1c57576bca444527 Oct 31 03:31:05.767021 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.766981 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:31:05.767592 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.767564 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:31:05.768087 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:05.768061 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:31:06.040233 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:06.037256 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" event={"ID":"5123bf1e-0ad6-4406-86b0-2b9a9e025a45","Type":"ContainerStarted","Data":"744a3ca1bfa7c5a9395b7d6b7b5d8b367ba6b48b2d8954fd459ae579b3c71f14"} Oct 31 03:31:06.040233 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:06.037320 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" event={"ID":"5123bf1e-0ad6-4406-86b0-2b9a9e025a45","Type":"ContainerStarted","Data":"8ab255d063fd92dd1de2fca11a413ce665318e934de0be896332fa49b1ff727e"} Oct 31 03:31:06.044545 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:06.044261 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" event={"ID":"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7","Type":"ContainerStarted","Data":"e24242cc752af656810ca6373322bafdec557f3a573129be7802d1166a376163"} Oct 31 03:31:06.044545 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:06.044321 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" event={"ID":"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7","Type":"ContainerStarted","Data":"4bafe6780a261405095d6783541ec7c61c313ecf175969ed1c57576bca444527"} Oct 31 03:31:06.065706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:06.065626 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" podStartSLOduration=2.06560773 podStartE2EDuration="2.06560773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:06.062459344 +0000 UTC m=+900.597154303" watchObservedRunningTime="2026-10-31 03:31:06.06560773 +0000 UTC m=+900.600302686" Oct 31 03:31:06.090895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:06.090834 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" podStartSLOduration=2.090814231 podStartE2EDuration="2.090814231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:06.085498058 +0000 UTC m=+900.620193018" watchObservedRunningTime="2026-10-31 03:31:06.090814231 +0000 UTC m=+900.625509186" Oct 31 03:31:06.381588 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:06.381492 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:06.382303 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:06.382273 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:31:06.382510 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:06.382479 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" podUID="5123bf1e-0ad6-4406-86b0-2b9a9e025a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:31:06.386151 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:06.386044 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:06.393013 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:06.392977 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 03:31:06.393174 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:06.393036 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 03:31:06.608706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:06.608664 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:07.377754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:07.377407 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:31:07.377754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:07.377477 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" podUID="5123bf1e-0ad6-4406-86b0-2b9a9e025a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:31:07.385789 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:07.385758 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 03:31:07.386238 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:07.386208 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 03:31:07.513299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:07.513256 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:08.381908 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:08.379144 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:31:08.381908 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:08.379211 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" podUID="5123bf1e-0ad6-4406-86b0-2b9a9e025a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:31:08.386339 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:08.386305 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 03:31:08.386734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:08.386360 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 03:31:08.484483 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:08.484442 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:09.384751 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:09.379759 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:31:09.384751 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:09.379825 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" podUID="5123bf1e-0ad6-4406-86b0-2b9a9e025a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:31:09.386114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:09.386078 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 03:31:09.386263 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:09.386136 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 03:31:09.417986 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:09.417840 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:10.382271 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:10.382219 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:31:10.382610 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:10.382573 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" podUID="5123bf1e-0ad6-4406-86b0-2b9a9e025a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:31:10.385909 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:10.385880 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 03:31:10.386026 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:10.385921 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 03:31:10.550486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:10.550432 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:11.390133 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:11.389555 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:31:11.390133 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:11.389629 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" podUID="5123bf1e-0ad6-4406-86b0-2b9a9e025a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:31:11.390988 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:11.390876 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 03:31:11.390988 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:11.390927 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 03:31:11.399828 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:11.399498 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:11.692931 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:11.692742 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:11.824998 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:11.824957 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:12.286209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:12.285951 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:12.377636 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:12.377596 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:31:12.378017 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:12.377980 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" podUID="5123bf1e-0ad6-4406-86b0-2b9a9e025a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:31:12.385712 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:12.385583 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 03:31:12.385712 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:12.385629 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 03:31:12.448551 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:12.448514 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:13.146786 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:13.137330 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:13.582457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:13.582431 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:13.583151 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:13.583120 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:13.584947 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:13.584923 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:31:13.594147 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:13.594116 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:13.595504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:13.595480 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:13.596785 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:13.596763 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:31:13.991656 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:13.991565 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:14.892152 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:14.892118 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:15.504241 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:15.504205 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:15.781696 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:15.781584 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:16.173662 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:16.173609 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:16.691173 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:16.691124 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:16.824584 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:16.824542 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:17.134526 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:31:17.134428 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:31:17.206771 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:17.206732 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:17.891365 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:17.891302 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:18.034957 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.034911 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:18.353782 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.353745 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js"] Oct 31 03:31:18.359937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.359894 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.373125 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.373097 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5aeec060-openshift-default-dockercfg-cg7rt\"" Oct 31 03:31:18.378486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.378459 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js"] Oct 31 03:31:18.452345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.452302 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:18.465091 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.465056 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-credential-socket\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.465280 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.465126 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-data\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.465280 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.465193 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-workload-certs\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.465280 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.465252 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-envoy\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.465511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.465360 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-podinfo\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.465511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.465457 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffk4r\" (UniqueName: \"kubernetes.io/projected/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-kube-api-access-ffk4r\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.465511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.465481 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-token\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.465966 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.465549 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istiod-ca-cert\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.465966 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.465610 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-workload-socket\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.567560 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.567520 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istiod-ca-cert\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.567771 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.567592 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-workload-socket\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.567771 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.567658 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-credential-socket\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.567771 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.567715 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-data\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.567771 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.567742 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-workload-certs\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.568012 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.567799 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-envoy\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.568012 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.567869 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-podinfo\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.568012 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.567990 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffk4r\" (UniqueName: \"kubernetes.io/projected/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-kube-api-access-ffk4r\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.568156 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.568026 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-token\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.568156 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.568053 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-workload-socket\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.568156 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.568120 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-credential-socket\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.568284 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.568247 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-data\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.568339 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.568285 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-workload-certs\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.568396 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.568375 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istiod-ca-cert\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.570424 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.570399 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-envoy\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.570656 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.570635 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-podinfo\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.583107 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.583078 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-token\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.583310 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.583279 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffk4r\" (UniqueName: \"kubernetes.io/projected/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-kube-api-access-ffk4r\") pod \"e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.608261 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.608184 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:18.680736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.680365 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:18.754230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:18.754193 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:19.279700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:19.279367 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:19.688595 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:19.686458 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:19.741611 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:19.741549 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:31:19.742142 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:19.742082 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:31:19.742399 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:19.742354 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:31:19.902297 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:19.901933 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:19.950092 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:19.950061 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js"] Oct 31 03:31:19.950378 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:19.950363 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s"] Oct 31 03:31:20.026984 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.026897 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s"] Oct 31 03:31:20.027171 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.027044 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.044640 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.044587 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:20.149699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.147771 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:20.166708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.166406 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:20.221764 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.192663 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.225097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.225058 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.228823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.228787 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.254705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.229252 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvcll\" (UniqueName: \"kubernetes.io/projected/35b39830-7aed-419d-b82f-fbf9c32128fa-kube-api-access-mvcll\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.255797 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.255761 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.256024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.256006 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.256143 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.256128 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.256273 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.256259 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.256375 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.256361 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35b39830-7aed-419d-b82f-fbf9c32128fa-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.522763 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.511510 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:31:20.577409 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.577360 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 31 03:31:20.599965 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.599921 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.607880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.607819 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvcll\" (UniqueName: \"kubernetes.io/projected/35b39830-7aed-419d-b82f-fbf9c32128fa-kube-api-access-mvcll\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.608174 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.608151 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.611699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.608325 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.666811 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.611895 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.668206 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.668178 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.668485 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.668463 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35b39830-7aed-419d-b82f-fbf9c32128fa-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.695337 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.664321 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.723906 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.668059 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.726920 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.726770 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:20.796339 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.796204 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.802795 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.801428 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35b39830-7aed-419d-b82f-fbf9c32128fa-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.802795 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.801583 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.802795 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.801782 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.802795 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.802623 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.872700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.857217 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.982648 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.873851 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvcll\" (UniqueName: \"kubernetes.io/projected/35b39830-7aed-419d-b82f-fbf9c32128fa-kube-api-access-mvcll\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.982849 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.875858 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:20.983222 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.895221 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.983341 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.961749 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:20.983427 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.961827 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:31:21.008510 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:20.982541 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:31:21.058663 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:21.008001 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:31:21.059217 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:21.033951 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:21.059643 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:21.052926 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:31:21.238702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:21.153943 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:31:21.238702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:21.227222 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:31:21.238702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:21.227312 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:31:21.238702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:21.166793 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:31:21.259713 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:21.180309 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:21.393758 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:21.387422 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:31:21.636903 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:21.636846 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:31:21.727893 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:21.727799 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:21.765014 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:21.742866 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:21.795719 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:21.795664 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:22.023024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.022937 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:31:22.274757 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.274080 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:22.274757 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.078762 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:22.305702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.078801 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:31:22.305702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.078821 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:31:22.305702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.104884 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:22.305702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.105137 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" podUID="f64d3e84-440a-4298-9230-ae9e0063c962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:22.305702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.192343 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" podUID="775a8940-cb22-4774-a43f-c6e7c5364354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:22.305702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.192549 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" podUID="80180652-fb4f-4976-956d-25a47344d581" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:22.305702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.248857 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:31:22.305702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.248903 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:31:22.305702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.026780 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" podUID="80180652-fb4f-4976-956d-25a47344d581" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:22.382702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.358131 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:22.643762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.643214 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:22.824364 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:31:22.676943 5490 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.039s" Oct 31 03:31:22.853975 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.853931 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" event={"ID":"bd68eadd-4bc5-4f8d-920f-e6e6babb8141","Type":"ContainerStarted","Data":"468f243937821bbea0ab0d872de6ee92e80b92e6ce30f00d189e396d445cea61"} Oct 31 03:31:22.854295 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.854275 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:31:22.854444 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.854432 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:31:22.854582 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.854569 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:31:22.870053 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.870017 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:31:22.870337 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.870316 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:31:22.870507 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.870492 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:31:22.882709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.804426 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:31:22.909751 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:22.759982 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:23.107854 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:23.107813 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:31:23.108132 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:23.108100 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" podUID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:23.325932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:23.285903 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:23.349442 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:23.284933 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:23.363812 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:23.363760 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:31:23.417705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:23.395416 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:31:23.417705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:23.409124 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:23.417705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:23.409492 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:31:23.417705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:23.410541 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:23.894219 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:23.894175 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Oct 31 03:31:23.970505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:23.909737 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Oct 31 03:31:24.040236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:24.040175 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:31:24.177884 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:24.140410 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:31:24.205319 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:24.186331 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:31:24.205319 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:24.186434 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:31:24.205319 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:24.186483 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:31:24.348975 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:24.348926 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:24.372390 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:24.357776 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:24.372390 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:24.364586 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:31:24.387147 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:24.385806 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:31:24.420827 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:24.409897 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:24.420827 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:24.413145 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:24.882390 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:24.422045 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:31:24.882390 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:24.439453 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:24.923698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:24.912793 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" event={"ID":"bd68eadd-4bc5-4f8d-920f-e6e6babb8141","Type":"ContainerStarted","Data":"fbb70630ac619caa4a7a1f605f253fe28952ac7f76bef8542569b503ea7c5c90"} Oct 31 03:31:25.053704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:25.052080 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:25.173054 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:25.164176 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:25.397822 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:25.397776 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:25.439558 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:25.434848 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:25.439917 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:25.439876 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:25.469832 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:25.441738 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:25.513714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:25.470804 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:25.527729 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:25.470784 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:31:25.527729 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:25.525799 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:31:25.894852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:25.892834 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:31:25.894852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:25.892949 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:26.137727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:26.113844 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:26.509701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:26.491986 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:26.525793 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:26.513762 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" podStartSLOduration=8.513735917 podStartE2EDuration="8.513735917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:24.780011912 +0000 UTC m=+919.314706894" watchObservedRunningTime="2026-10-31 03:31:26.513735917 +0000 UTC m=+921.048430873" Oct 31 03:31:26.551077 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:26.551030 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:26.553524 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:26.553489 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:31:26.555574 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:26.555544 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:31:26.559694 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:26.556078 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:31:26.560423 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:26.560393 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:31:26.624999 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:26.624900 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:31:26.993185 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:26.983776 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded" Oct 31 03:31:27.114931 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.114869 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:27.116134 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.116087 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:27.299856 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:31:27.284512 5490 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.391s" Oct 31 03:31:27.308424 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.308395 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s"] Oct 31 03:31:27.308698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.308668 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:27.308832 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.308813 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" event={"ID":"35b39830-7aed-419d-b82f-fbf9c32128fa","Type":"ContainerStarted","Data":"4734d76071dc676eb9379ce3a8f4c64caf9815776fb9d54c360269bbac405665"} Oct 31 03:31:27.308933 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.308922 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:27.309059 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.309051 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:31:27.310109 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.310041 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded" Oct 31 03:31:27.311838 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.311817 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:31:27.393942 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.393902 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:27.426987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.406045 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:31:27.519273 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.519242 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:31:27.587767 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.553954 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:27.675871 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.666401 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:31:27.709761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.709454 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:27.713142 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.713110 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:27.735087 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.728361 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:27.781021 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.780820 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:31:27.781366 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.781349 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:31:27.781721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.781702 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:31:27.783091 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.783068 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:31:27.783199 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.783115 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:31:27.865592 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:27.865506 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" event={"ID":"35b39830-7aed-419d-b82f-fbf9c32128fa","Type":"ContainerStarted","Data":"a6f7cb0b21e695fa4ba2d3e897c29b29f2eec10e3f9950815dfb298278c04f89"} Oct 31 03:31:28.180594 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:28.180558 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:28.182322 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:28.182282 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 03:31:28.182477 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:28.182345 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" podUID="35b39830-7aed-419d-b82f-fbf9c32128fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 03:31:28.593903 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:28.593807 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:29.181462 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:29.181416 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 03:31:29.181668 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:29.181492 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" podUID="35b39830-7aed-419d-b82f-fbf9c32128fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 03:31:29.189108 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:29.189016 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:29.355712 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:29.354411 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:30.037534 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:30.037425 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:30.181366 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:30.181332 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 03:31:30.181544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:30.181385 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" podUID="35b39830-7aed-419d-b82f-fbf9c32128fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 03:31:30.213465 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:30.213429 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:30.750713 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:30.750610 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:30.903367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:30.903111 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:31.052466 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:31.052288 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:31.184704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:31.184146 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 03:31:31.184704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:31.184209 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" podUID="35b39830-7aed-419d-b82f-fbf9c32128fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 03:31:31.223846 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:31.216385 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:31.378758 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:31.378061 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:32.186718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:32.186014 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 03:31:32.186718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:32.186075 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" podUID="35b39830-7aed-419d-b82f-fbf9c32128fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 03:31:32.443510 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:32.443318 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:32.606771 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:32.606728 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:32.759768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:32.758813 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:33.187954 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:33.187916 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 03:31:33.188542 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:33.188512 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" podUID="35b39830-7aed-419d-b82f-fbf9c32128fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 03:31:33.503241 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:33.503147 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:34.181806 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:34.181762 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 31 03:31:34.182122 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:34.182084 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" podUID="35b39830-7aed-419d-b82f-fbf9c32128fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 31 03:31:34.440955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:34.440872 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:34.634165 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:34.634105 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:35.284192 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:35.284152 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:35.337973 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:35.332229 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:35.337973 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:35.333709 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:35.337973 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:35.336727 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:31:35.364260 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:35.358796 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" podStartSLOduration=16.35877733 podStartE2EDuration="16.35877733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:27.992093656 +0000 UTC m=+922.526788611" watchObservedRunningTime="2026-10-31 03:31:35.35877733 +0000 UTC m=+929.893472285" Oct 31 03:31:35.880009 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:35.879972 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg"] Oct 31 03:31:35.886172 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:35.886136 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" Oct 31 03:31:35.896019 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:35.895987 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg"] Oct 31 03:31:35.896892 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:35.896863 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5aeec060-serving-cert\"" Oct 31 03:31:35.963830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:35.963792 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c52e703-346c-4b26-89b7-a8a8954ad15f-maas-api-tls\") pod \"maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg\" (UID: \"1c52e703-346c-4b26-89b7-a8a8954ad15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" Oct 31 03:31:35.964037 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:35.963919 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rslfs\" (UniqueName: \"kubernetes.io/projected/1c52e703-346c-4b26-89b7-a8a8954ad15f-kube-api-access-rslfs\") pod \"maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg\" (UID: \"1c52e703-346c-4b26-89b7-a8a8954ad15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" Oct 31 03:31:35.964037 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:35.964031 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c52e703-346c-4b26-89b7-a8a8954ad15f-openshift-service-ca\") pod \"maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg\" (UID: \"1c52e703-346c-4b26-89b7-a8a8954ad15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" Oct 31 03:31:35.964169 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:35.964141 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c52e703-346c-4b26-89b7-a8a8954ad15f-metrics-certs\") pod \"maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg\" (UID: \"1c52e703-346c-4b26-89b7-a8a8954ad15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" Oct 31 03:31:35.978759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:35.978718 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:36.068124 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:36.066227 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c52e703-346c-4b26-89b7-a8a8954ad15f-maas-api-tls\") pod \"maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg\" (UID: \"1c52e703-346c-4b26-89b7-a8a8954ad15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" Oct 31 03:31:36.068124 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:36.066348 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rslfs\" (UniqueName: \"kubernetes.io/projected/1c52e703-346c-4b26-89b7-a8a8954ad15f-kube-api-access-rslfs\") pod \"maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg\" (UID: \"1c52e703-346c-4b26-89b7-a8a8954ad15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" Oct 31 03:31:36.068124 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:36.066958 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c52e703-346c-4b26-89b7-a8a8954ad15f-openshift-service-ca\") pod \"maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg\" (UID: \"1c52e703-346c-4b26-89b7-a8a8954ad15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" Oct 31 03:31:36.068124 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:36.067033 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c52e703-346c-4b26-89b7-a8a8954ad15f-metrics-certs\") pod \"maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg\" (UID: \"1c52e703-346c-4b26-89b7-a8a8954ad15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" Oct 31 03:31:36.069605 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:36.069574 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c52e703-346c-4b26-89b7-a8a8954ad15f-openshift-service-ca\") pod \"maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg\" (UID: \"1c52e703-346c-4b26-89b7-a8a8954ad15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" Oct 31 03:31:36.074908 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:36.074880 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c52e703-346c-4b26-89b7-a8a8954ad15f-metrics-certs\") pod \"maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg\" (UID: \"1c52e703-346c-4b26-89b7-a8a8954ad15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" Oct 31 03:31:36.077489 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:36.077465 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c52e703-346c-4b26-89b7-a8a8954ad15f-maas-api-tls\") pod \"maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg\" (UID: \"1c52e703-346c-4b26-89b7-a8a8954ad15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" Oct 31 03:31:36.077888 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:36.077862 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rslfs\" (UniqueName: \"kubernetes.io/projected/1c52e703-346c-4b26-89b7-a8a8954ad15f-kube-api-access-rslfs\") pod \"maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg\" (UID: \"1c52e703-346c-4b26-89b7-a8a8954ad15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" Oct 31 03:31:36.110041 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:36.110001 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:36.237613 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:36.237576 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" Oct 31 03:31:36.238131 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:36.238097 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:36.611520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:36.611160 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg"] Oct 31 03:31:36.619315 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:31:36.618601 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c52e703_346c_4b26_89b7_a8a8954ad15f.slice/crio-28c0611b40b79b990999f9af8222c373a03f18b167d1694d51d628c2001c5b6a WatchSource:0}: Error finding container 28c0611b40b79b990999f9af8222c373a03f18b167d1694d51d628c2001c5b6a: Status 404 returned error can't find the container with id 28c0611b40b79b990999f9af8222c373a03f18b167d1694d51d628c2001c5b6a Oct 31 03:31:36.620188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:36.620165 5490 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:31:36.885657 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:36.885573 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:36.975523 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:36.975486 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" event={"ID":"1c52e703-346c-4b26-89b7-a8a8954ad15f","Type":"ContainerStarted","Data":"28c0611b40b79b990999f9af8222c373a03f18b167d1694d51d628c2001c5b6a"} Oct 31 03:31:37.586234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:37.567984 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:37.985945 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:37.985906 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" event={"ID":"1c52e703-346c-4b26-89b7-a8a8954ad15f","Type":"ContainerStarted","Data":"52e8ae698db6291d3d53ee34cfbb284533895c827e5a324a568c234e6b1d1a85"} Oct 31 03:31:37.986431 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:37.986006 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" Oct 31 03:31:38.007598 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:38.007534 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" podStartSLOduration=2.39616736 podStartE2EDuration="3.007519206s" totalImagesPullingTime="611.351846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:38.002878202 +0000 UTC m=+932.537573169" watchObservedRunningTime="2026-10-31 03:31:38.007519206 +0000 UTC m=+932.542214160" Oct 31 03:31:38.396446 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:38.396359 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:38.539557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:38.539516 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:38.855753 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:38.855715 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:39.227350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:39.227307 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:40.068921 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.068878 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:40.595365 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.595330 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg"] Oct 31 03:31:40.610037 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.610007 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" Oct 31 03:31:40.611666 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.611637 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg"] Oct 31 03:31:40.616021 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.615996 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1c45857e-serving-cert\"" Oct 31 03:31:40.683668 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.683635 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:40.801322 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.801277 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8zls\" (UniqueName: \"kubernetes.io/projected/3b219805-f5a8-4f72-a014-0f629263010a-kube-api-access-w8zls\") pod \"maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg\" (UID: \"3b219805-f5a8-4f72-a014-0f629263010a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" Oct 31 03:31:40.801560 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.801537 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b219805-f5a8-4f72-a014-0f629263010a-openshift-service-ca\") pod \"maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg\" (UID: \"3b219805-f5a8-4f72-a014-0f629263010a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" Oct 31 03:31:40.801613 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.801596 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b219805-f5a8-4f72-a014-0f629263010a-maas-api-tls\") pod \"maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg\" (UID: \"3b219805-f5a8-4f72-a014-0f629263010a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" Oct 31 03:31:40.801790 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.801766 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b219805-f5a8-4f72-a014-0f629263010a-metrics-certs\") pod \"maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg\" (UID: \"3b219805-f5a8-4f72-a014-0f629263010a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" Oct 31 03:31:40.811109 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.811073 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:40.903913 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.903889 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b219805-f5a8-4f72-a014-0f629263010a-maas-api-tls\") pod \"maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg\" (UID: \"3b219805-f5a8-4f72-a014-0f629263010a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" Oct 31 03:31:40.904075 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.903950 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b219805-f5a8-4f72-a014-0f629263010a-metrics-certs\") pod \"maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg\" (UID: \"3b219805-f5a8-4f72-a014-0f629263010a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" Oct 31 03:31:40.904075 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.904065 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8zls\" (UniqueName: \"kubernetes.io/projected/3b219805-f5a8-4f72-a014-0f629263010a-kube-api-access-w8zls\") pod \"maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg\" (UID: \"3b219805-f5a8-4f72-a014-0f629263010a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" Oct 31 03:31:40.904230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.904218 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b219805-f5a8-4f72-a014-0f629263010a-openshift-service-ca\") pod \"maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg\" (UID: \"3b219805-f5a8-4f72-a014-0f629263010a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" Oct 31 03:31:40.904854 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.904832 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b219805-f5a8-4f72-a014-0f629263010a-openshift-service-ca\") pod \"maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg\" (UID: \"3b219805-f5a8-4f72-a014-0f629263010a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" Oct 31 03:31:40.906512 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.906487 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b219805-f5a8-4f72-a014-0f629263010a-maas-api-tls\") pod \"maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg\" (UID: \"3b219805-f5a8-4f72-a014-0f629263010a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" Oct 31 03:31:40.906582 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.906560 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b219805-f5a8-4f72-a014-0f629263010a-metrics-certs\") pod \"maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg\" (UID: \"3b219805-f5a8-4f72-a014-0f629263010a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" Oct 31 03:31:40.917263 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.917231 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8zls\" (UniqueName: \"kubernetes.io/projected/3b219805-f5a8-4f72-a014-0f629263010a-kube-api-access-w8zls\") pod \"maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg\" (UID: \"3b219805-f5a8-4f72-a014-0f629263010a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" Oct 31 03:31:40.930060 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.930035 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" Oct 31 03:31:40.936836 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:40.936810 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:41.315782 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:41.311484 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg"] Oct 31 03:31:41.707991 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:41.707951 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:41.859770 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:41.859333 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:42.075161 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:42.075113 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" event={"ID":"3b219805-f5a8-4f72-a014-0f629263010a","Type":"ContainerStarted","Data":"55da22283ec74ad49dd74a9688cd2a201081963c19c6d7d8b75f077a17f5c0dd"} Oct 31 03:31:42.075383 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:42.075185 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" event={"ID":"3b219805-f5a8-4f72-a014-0f629263010a","Type":"ContainerStarted","Data":"f3f26c5c495fc9142a1083894b3f4eff70c8380d9c4a3e0c4a113c50449f90b5"} Oct 31 03:31:42.075815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:42.075790 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" Oct 31 03:31:42.116249 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:42.116198 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" podStartSLOduration=1.683437168 podStartE2EDuration="2.116182595s" totalImagesPullingTime="432.745427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:42.111631351 +0000 UTC m=+936.646326318" watchObservedRunningTime="2026-10-31 03:31:42.116182595 +0000 UTC m=+936.650877550" Oct 31 03:31:42.119084 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:42.119043 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:42.337332 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:42.337232 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:42.664619 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:42.664584 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:43.031501 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:43.031411 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:43.171759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:43.171722 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:43.310889 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:43.310807 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:43.501165 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:43.501132 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:44.024192 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:44.024161 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" Oct 31 03:31:44.312728 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:44.311637 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:44.910815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:44.910773 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:45.853047 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:45.852736 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:46.661700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:46.659093 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:47.135161 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:31:47.135066 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:31:47.654500 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:47.653479 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:48.091112 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:48.091027 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" Oct 31 03:31:48.373539 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:48.373401 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:49.271899 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:49.271852 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:49.407941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:49.407410 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:49.993819 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:49.993238 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:50.807262 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:50.807220 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:51.494189 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.494156 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m"] Oct 31 03:31:51.498764 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.498743 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.509584 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.509554 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m"] Oct 31 03:31:51.509911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.509891 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 03:31:51.510058 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.509947 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 03:31:51.510058 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.509982 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 03:31:51.511748 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.511214 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-hwjnd\"" Oct 31 03:31:51.567965 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.567921 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g8sv\" (UniqueName: \"kubernetes.io/projected/decb2697-571c-48e2-88b3-84a03dfbd80a-kube-api-access-2g8sv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.567965 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.567968 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.568223 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.568084 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.568223 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.568109 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.568223 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.568140 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.568395 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.568338 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.568498 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.568476 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/decb2697-571c-48e2-88b3-84a03dfbd80a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.631756 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.631715 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:51.669738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.669655 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g8sv\" (UniqueName: \"kubernetes.io/projected/decb2697-571c-48e2-88b3-84a03dfbd80a-kube-api-access-2g8sv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.669931 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.669766 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.669931 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.669866 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.669931 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.669906 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.670121 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.669946 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.670121 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.670079 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.670243 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.670222 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/decb2697-571c-48e2-88b3-84a03dfbd80a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.670849 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.670811 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.672656 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.672610 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.679876 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.672851 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/decb2697-571c-48e2-88b3-84a03dfbd80a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.679876 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.672943 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.679876 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.672977 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.679876 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.678456 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g8sv\" (UniqueName: \"kubernetes.io/projected/decb2697-571c-48e2-88b3-84a03dfbd80a-kube-api-access-2g8sv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.679876 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.679040 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7x7m\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:51.824967 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:51.824888 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:31:52.248469 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:52.238590 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m"] Oct 31 03:31:52.256308 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:31:52.256270 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddecb2697_571c_48e2_88b3_84a03dfbd80a.slice/crio-ac24b9074865b1e8b02dd6467baa265e601bd129deb3bf2fff9a64acaf1cf13a WatchSource:0}: Error finding container ac24b9074865b1e8b02dd6467baa265e601bd129deb3bf2fff9a64acaf1cf13a: Status 404 returned error can't find the container with id ac24b9074865b1e8b02dd6467baa265e601bd129deb3bf2fff9a64acaf1cf13a Oct 31 03:31:52.633917 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:52.633596 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:53.300248 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:53.300026 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" event={"ID":"decb2697-571c-48e2-88b3-84a03dfbd80a","Type":"ContainerStarted","Data":"ac24b9074865b1e8b02dd6467baa265e601bd129deb3bf2fff9a64acaf1cf13a"} Oct 31 03:31:54.296000 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:54.295813 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": context deadline exceeded" Oct 31 03:31:54.417965 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:54.416849 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:54.516244 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:54.504760 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:54.883439 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:54.883389 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:55.047825 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:55.047773 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:55.119530 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:55.119483 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:55.351898 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:55.343927 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:55.370955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:55.358155 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:55.386786 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:55.373278 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:31:55.438305 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:55.429050 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:55.438305 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:55.429489 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:31:55.448786 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:55.445115 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:55.537712 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:55.533836 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:55.537712 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:55.534477 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:31:55.886827 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:55.886770 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:31:56.145882 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:56.145767 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:31:56.147427 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:56.147391 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:31:56.316703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:56.286993 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:31:56.316703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:56.287061 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:31:56.316703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:56.287099 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:31:56.378750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:56.363516 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:56.394623 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:56.393789 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:31:56.394854 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:56.378017 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:56.395148 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:56.395125 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:31:56.395288 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:56.378060 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:31:56.462787 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:56.454194 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:56.476188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:56.463450 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:31:56.476188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:56.465312 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:56.545967 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:56.545918 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:31:56.785736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:56.773242 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:56.888452 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:56.888403 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:57.159803 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:57.153290 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:57.178759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:57.162951 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded" Oct 31 03:31:57.255825 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:57.194245 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:57.353734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:57.351242 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:31:57.406787 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:57.406734 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:57.494549 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:57.422831 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:57.532851 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:57.422906 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:57.533153 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:57.494062 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:57.533312 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:57.513287 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:31:58.014175 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:58.014109 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:31:58.046709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:58.032793 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:58.169717 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:58.151759 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:58.169717 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:58.167771 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:58.239037 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:58.239000 5490 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-5kbd8 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.11:9443/health\": context deadline exceeded" start-of-body= Oct 31 03:31:58.300782 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:58.277016 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" podUID="8bae2714-9c1b-44db-a6ef-b0b008cf1fa2" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.133.0.11:9443/health\": context deadline exceeded" Oct 31 03:31:58.300975 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:58.275797 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:58.307702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:58.304342 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:58.580755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:58.577057 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:31:58.762703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:58.762319 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:31:58.762703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:58.762381 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" podUID="cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:59.154368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:59.154324 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:31:59.156203 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:59.156168 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:31:59.185703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:59.182411 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:31:59.186426 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:59.186385 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:31:59.201504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:59.201462 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:31:59.201927 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:59.201902 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:59.274820 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:59.201835 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" podUID="cdf49e86-d2b1-4888-8475-8a85dd69a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:59.296887 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:59.267939 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:31:59.314362 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:59.314308 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:31:59.332713 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:59.317985 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.451964999s: [/var/lib/containers/storage/overlay/0bc1ff134555d413f9bdaf5b98937c5c19ad375dc8315a833c42ea1e4c521e7e/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf_4cf50264-0d76-46ca-b2cb-998ae3bbeb1b/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:31:59.563700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:59.563590 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:31:59.629826 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:59.629769 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" podUID="60771f19-56cc-40a1-a27c-b09f23ac9903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:59.668700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:59.632955 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:31:59.810476 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:59.810422 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:31:59.970860 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:59.970810 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:31:59.991707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:31:59.978914 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.170747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:00.162467 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.170747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:00.163811 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:32:00.189085 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:00.188069 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.189085 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:00.188603 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.230921 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:00.230823 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:32:00.231177 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:00.231146 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" podUID="5123bf1e-0ad6-4406-86b0-2b9a9e025a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.284587 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:00.283787 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.318261 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:00.298174 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.344270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:00.343258 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.351307 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:00.350969 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:32:00.351307 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:00.351035 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" podUID="113276f1-a2bd-49bd-843e-22146959715e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.426403 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:00.426152 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.730119 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:00.730059 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.821814 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:00.814479 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:00.955723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:00.953201 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Oct 31 03:32:01.172711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:01.162661 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:32:01.187711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:01.179813 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:32:01.197704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:01.191615 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:01.197704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:01.191788 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:01.300715 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:01.298309 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:32:01.313703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:01.312775 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:01.348728 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:01.345852 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:01.489724 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:01.489529 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:01.733694 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:01.733629 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:01.753234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:01.736777 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:01.841703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:01.809783 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:01.849718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:01.821731 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:32:02.228355 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:02.223823 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:32:02.239702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:02.232466 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:02.289597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:02.233505 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:02.484833 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:02.458750 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:32:02.484833 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:02.462715 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:02.484833 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:02.464378 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:32:02.583707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:02.582791 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:02.894117 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:02.894013 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:32:02.914697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:02.914630 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:02.997703 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:32:02.981250 5490 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.403s" Oct 31 03:32:03.264733 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:03.234414 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:03.567496 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:03.341320 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:32:03.569801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:03.344517 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:03.571514 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:03.478720 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:03.572305 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:03.478759 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:32:03.573043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:03.481774 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:03.573196 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:03.484059 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:32:03.580711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:03.522380 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:03.581066 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:03.579874 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-k9bcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:32:03.581302 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:03.581275 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" podUID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:03.726557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:03.718873 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:32:03.828779 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:03.798346 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Oct 31 03:32:04.291916 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:04.224607 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Oct 31 03:32:04.291916 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:04.226602 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:32:04.291916 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:04.247819 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:04.371933 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:04.371878 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": context deadline exceeded" Oct 31 03:32:04.582772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:04.581201 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:32:04.582772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:04.581968 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:32:04.582772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:04.582708 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:32:04.589473 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:04.583704 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:32:04.610096 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:04.604372 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:32:04.626130 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:04.610597 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:32:04.775713 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:04.730758 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg"] Oct 31 03:32:04.835816 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:04.744622 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:04.852911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:04.784344 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" podUID="1c52e703-346c-4b26-89b7-a8a8954ad15f" containerName="maas-api" containerID="cri-o://52e8ae698db6291d3d53ee34cfbb284533895c827e5a324a568c234e6b1d1a85" gracePeriod=30 Oct 31 03:32:04.874702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:04.836226 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:05.184798 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:05.181779 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:05.242818 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:05.232259 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:32:05.278850 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:05.251803 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:05.317002 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:05.309771 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:32:05.397699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:05.397646 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:32:05.432825 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:05.432788 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:32:05.611082 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:05.589427 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:32:05.611082 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:05.594135 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:32:05.611082 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:05.607369 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:05.627394 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:05.625138 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:32:05.627394 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:05.626941 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:05.634706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:05.628920 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:32:05.681150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:05.681117 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:32:05.872358 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:05.872262 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:05.897069 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:05.897041 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:32:05.897334 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:05.897307 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:32:05.908157 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:05.908132 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js"] Oct 31 03:32:05.908603 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:05.908574 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" podUID="bd68eadd-4bc5-4f8d-920f-e6e6babb8141" containerName="istio-proxy" containerID="cri-o://fbb70630ac619caa4a7a1f605f253fe28952ac7f76bef8542569b503ea7c5c90" gracePeriod=30 Oct 31 03:32:06.545182 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:06.545139 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:06.589363 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:06.589322 5490 generic.go:356] "Generic (PLEG): container finished" podID="1c52e703-346c-4b26-89b7-a8a8954ad15f" containerID="52e8ae698db6291d3d53ee34cfbb284533895c827e5a324a568c234e6b1d1a85" exitCode=0 Oct 31 03:32:06.589756 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:06.589728 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" event={"ID":"1c52e703-346c-4b26-89b7-a8a8954ad15f","Type":"ContainerDied","Data":"52e8ae698db6291d3d53ee34cfbb284533895c827e5a324a568c234e6b1d1a85"} Oct 31 03:32:06.614751 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:06.614716 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" Oct 31 03:32:06.687793 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:06.687763 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c52e703-346c-4b26-89b7-a8a8954ad15f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c52e703-346c-4b26-89b7-a8a8954ad15f-metrics-certs\") pod \"1c52e703-346c-4b26-89b7-a8a8954ad15f\" (UID: \"1c52e703-346c-4b26-89b7-a8a8954ad15f\") " Oct 31 03:32:06.688458 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:06.688437 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c52e703-346c-4b26-89b7-a8a8954ad15f-kube-api-access-rslfs\" (UniqueName: \"kubernetes.io/projected/1c52e703-346c-4b26-89b7-a8a8954ad15f-kube-api-access-rslfs\") pod \"1c52e703-346c-4b26-89b7-a8a8954ad15f\" (UID: \"1c52e703-346c-4b26-89b7-a8a8954ad15f\") " Oct 31 03:32:06.688722 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:06.688701 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c52e703-346c-4b26-89b7-a8a8954ad15f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c52e703-346c-4b26-89b7-a8a8954ad15f-openshift-service-ca\") pod \"1c52e703-346c-4b26-89b7-a8a8954ad15f\" (UID: \"1c52e703-346c-4b26-89b7-a8a8954ad15f\") " Oct 31 03:32:06.689033 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:06.689015 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c52e703-346c-4b26-89b7-a8a8954ad15f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c52e703-346c-4b26-89b7-a8a8954ad15f-maas-api-tls\") pod \"1c52e703-346c-4b26-89b7-a8a8954ad15f\" (UID: \"1c52e703-346c-4b26-89b7-a8a8954ad15f\") " Oct 31 03:32:06.689339 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:06.689317 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c52e703-346c-4b26-89b7-a8a8954ad15f-openshift-service-ca" pod "1c52e703-346c-4b26-89b7-a8a8954ad15f" (UID: "1c52e703-346c-4b26-89b7-a8a8954ad15f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:32:06.690075 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:06.690051 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c52e703-346c-4b26-89b7-a8a8954ad15f-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:06.697404 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:06.697377 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c52e703-346c-4b26-89b7-a8a8954ad15f-maas-api-tls" pod "1c52e703-346c-4b26-89b7-a8a8954ad15f" (UID: "1c52e703-346c-4b26-89b7-a8a8954ad15f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:32:06.703394 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:06.703230 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c52e703-346c-4b26-89b7-a8a8954ad15f-kube-api-access-rslfs" pod "1c52e703-346c-4b26-89b7-a8a8954ad15f" (UID: "1c52e703-346c-4b26-89b7-a8a8954ad15f"). InnerVolumeSpecName "kube-api-access-rslfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:32:06.703661 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:06.703635 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c52e703-346c-4b26-89b7-a8a8954ad15f-metrics-certs" pod "1c52e703-346c-4b26-89b7-a8a8954ad15f" (UID: "1c52e703-346c-4b26-89b7-a8a8954ad15f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:32:06.793362 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:06.792803 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c52e703-346c-4b26-89b7-a8a8954ad15f-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:06.793362 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:06.792846 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c52e703-346c-4b26-89b7-a8a8954ad15f-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:06.793362 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:06.792863 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rslfs\" (UniqueName: \"kubernetes.io/projected/1c52e703-346c-4b26-89b7-a8a8954ad15f-kube-api-access-rslfs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:07.015822 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:07.015766 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:07.197439 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:07.197347 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:07.614700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:07.611111 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" event={"ID":"1c52e703-346c-4b26-89b7-a8a8954ad15f","Type":"ContainerDied","Data":"28c0611b40b79b990999f9af8222c373a03f18b167d1694d51d628c2001c5b6a"} Oct 31 03:32:07.614700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:07.611133 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg" Oct 31 03:32:07.614700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:07.611243 5490 scope.go:122] "RemoveContainer" containerID="52e8ae698db6291d3d53ee34cfbb284533895c827e5a324a568c234e6b1d1a85" Oct 31 03:32:07.665956 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:07.665891 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg"] Oct 31 03:32:07.672197 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:07.671873 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5aeec060-f6b57b7dd-hwsfg"] Oct 31 03:32:08.090926 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:08.090856 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:08.373463 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:08.373199 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:09.009528 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:09.009490 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:09.590750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:09.590056 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c52e703-346c-4b26-89b7-a8a8954ad15f" path="/var/lib/kubelet/pods/1c52e703-346c-4b26-89b7-a8a8954ad15f/volumes" Oct 31 03:32:09.889789 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:09.889629 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:10.040204 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:10.040119 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:10.198286 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:10.198233 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:11.099112 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:11.097806 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:11.396772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:11.386915 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:32:11.396772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:11.387004 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:32:11.552442 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:11.552318 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:11.686270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:11.686223 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" event={"ID":"decb2697-571c-48e2-88b3-84a03dfbd80a","Type":"ContainerStarted","Data":"be2efe71c5d89dd064bd84b0cbaeb4f452545b619be5ec4cbc00703b6e0dc887"} Oct 31 03:32:11.686518 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:11.686495 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:32:11.708111 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:11.708073 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:11.724738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:11.724233 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" podStartSLOduration=1.6019164529999999 podStartE2EDuration="20.724211145s" totalImagesPullingTime="19.122294692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:11.714455702 +0000 UTC m=+966.249150659" watchObservedRunningTime="2026-10-31 03:32:11.724211145 +0000 UTC m=+966.258906102" Oct 31 03:32:11.879701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:11.879470 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:12.099750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:12.093490 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:12.282699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:12.271103 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:12.715642 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:12.715608 5490 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-t7x7m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:32:12.715972 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:12.715941 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" podUID="decb2697-571c-48e2-88b3-84a03dfbd80a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:32:12.923891 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:12.923838 5490 patch_prober.go:29] interesting pod/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:32:12.924074 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:12.923940 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" podUID="bd68eadd-4bc5-4f8d-920f-e6e6babb8141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:13.421059 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:13.421020 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:14.318745 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.318716 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:32:14.352730 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.352640 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:14.417253 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.417219 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-podinfo\") pod \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " Oct 31 03:32:14.417468 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.417331 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-credential-socket\") pod \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " Oct 31 03:32:14.417468 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.417363 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-kube-api-access-ffk4r\" (UniqueName: \"kubernetes.io/projected/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-kube-api-access-ffk4r\") pod \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " Oct 31 03:32:14.417601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.417473 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-data\") pod \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " Oct 31 03:32:14.417662 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.417624 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-workload-certs\") pod \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " Oct 31 03:32:14.417996 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.417968 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-token\" (UniqueName: \"kubernetes.io/projected/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-token\") pod \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " Oct 31 03:32:14.418167 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.418147 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istiod-ca-cert\") pod \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " Oct 31 03:32:14.418265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.418244 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-envoy\") pod \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " Oct 31 03:32:14.418333 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.418299 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-workload-socket\") pod \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\" (UID: \"bd68eadd-4bc5-4f8d-920f-e6e6babb8141\") " Oct 31 03:32:14.418483 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.417967 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-credential-socket" pod "bd68eadd-4bc5-4f8d-920f-e6e6babb8141" (UID: "bd68eadd-4bc5-4f8d-920f-e6e6babb8141"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:32:14.418601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.418482 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-data" pod "bd68eadd-4bc5-4f8d-920f-e6e6babb8141" (UID: "bd68eadd-4bc5-4f8d-920f-e6e6babb8141"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:32:14.418601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.418546 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-workload-certs" pod "bd68eadd-4bc5-4f8d-920f-e6e6babb8141" (UID: "bd68eadd-4bc5-4f8d-920f-e6e6babb8141"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:32:14.419080 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.419047 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istiod-ca-cert" pod "bd68eadd-4bc5-4f8d-920f-e6e6babb8141" (UID: "bd68eadd-4bc5-4f8d-920f-e6e6babb8141"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:32:14.419400 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.419374 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:14.419488 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.419408 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:14.419488 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.419427 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:14.419488 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.419442 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:14.419731 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.419706 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-workload-socket" pod "bd68eadd-4bc5-4f8d-920f-e6e6babb8141" (UID: "bd68eadd-4bc5-4f8d-920f-e6e6babb8141"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:32:14.427392 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.427182 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-token" pod "bd68eadd-4bc5-4f8d-920f-e6e6babb8141" (UID: "bd68eadd-4bc5-4f8d-920f-e6e6babb8141"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:32:14.427392 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.427236 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-kube-api-access-ffk4r" pod "bd68eadd-4bc5-4f8d-920f-e6e6babb8141" (UID: "bd68eadd-4bc5-4f8d-920f-e6e6babb8141"). InnerVolumeSpecName "kube-api-access-ffk4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:32:14.428140 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.427401 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-podinfo" pod "bd68eadd-4bc5-4f8d-920f-e6e6babb8141" (UID: "bd68eadd-4bc5-4f8d-920f-e6e6babb8141"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:32:14.462227 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.462187 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-envoy" pod "bd68eadd-4bc5-4f8d-920f-e6e6babb8141" (UID: "bd68eadd-4bc5-4f8d-920f-e6e6babb8141"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:32:14.520982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.520953 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:14.520982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.520979 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:14.520982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.520990 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:14.521213 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.520999 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:14.521213 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.521009 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffk4r\" (UniqueName: \"kubernetes.io/projected/bd68eadd-4bc5-4f8d-920f-e6e6babb8141-kube-api-access-ffk4r\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:14.732433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.732384 5490 generic.go:356] "Generic (PLEG): container finished" podID="bd68eadd-4bc5-4f8d-920f-e6e6babb8141" containerID="fbb70630ac619caa4a7a1f605f253fe28952ac7f76bef8542569b503ea7c5c90" exitCode=0 Oct 31 03:32:14.732626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.732440 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" Oct 31 03:32:14.732626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.732467 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" event={"ID":"bd68eadd-4bc5-4f8d-920f-e6e6babb8141","Type":"ContainerDied","Data":"fbb70630ac619caa4a7a1f605f253fe28952ac7f76bef8542569b503ea7c5c90"} Oct 31 03:32:14.732626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.732539 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js" event={"ID":"bd68eadd-4bc5-4f8d-920f-e6e6babb8141","Type":"ContainerDied","Data":"468f243937821bbea0ab0d872de6ee92e80b92e6ce30f00d189e396d445cea61"} Oct 31 03:32:14.732626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.732568 5490 scope.go:122] "RemoveContainer" containerID="fbb70630ac619caa4a7a1f605f253fe28952ac7f76bef8542569b503ea7c5c90" Oct 31 03:32:14.744642 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.744617 5490 scope.go:122] "RemoveContainer" containerID="fbb70630ac619caa4a7a1f605f253fe28952ac7f76bef8542569b503ea7c5c90" Oct 31 03:32:14.744994 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:32:14.744964 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb70630ac619caa4a7a1f605f253fe28952ac7f76bef8542569b503ea7c5c90\": container with ID starting with fbb70630ac619caa4a7a1f605f253fe28952ac7f76bef8542569b503ea7c5c90 not found: ID does not exist" containerID="fbb70630ac619caa4a7a1f605f253fe28952ac7f76bef8542569b503ea7c5c90" Oct 31 03:32:14.745087 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.745005 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb70630ac619caa4a7a1f605f253fe28952ac7f76bef8542569b503ea7c5c90"} err="failed to get container status \"fbb70630ac619caa4a7a1f605f253fe28952ac7f76bef8542569b503ea7c5c90\": rpc error: code = NotFound desc = could not find container \"fbb70630ac619caa4a7a1f605f253fe28952ac7f76bef8542569b503ea7c5c90\": container with ID starting with fbb70630ac619caa4a7a1f605f253fe28952ac7f76bef8542569b503ea7c5c90 not found: ID does not exist" Oct 31 03:32:14.756467 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.756439 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js"] Oct 31 03:32:14.765526 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:14.765501 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5aeec060-openshift-default-849947dcb7-rg7js"] Oct 31 03:32:15.147231 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:15.147146 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:15.582705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:15.580291 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd68eadd-4bc5-4f8d-920f-e6e6babb8141" path="/var/lib/kubelet/pods/bd68eadd-4bc5-4f8d-920f-e6e6babb8141/volumes" Oct 31 03:32:15.712529 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:15.712493 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:32:15.861268 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:15.861147 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:15.993335 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:15.993263 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:16.679606 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:16.679566 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:17.133502 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:32:17.133419 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:32:17.413470 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:17.412886 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:18.086209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:18.086176 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:18.217836 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:18.217564 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:18.368075 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:18.367987 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:19.068111 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:19.068071 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:19.604991 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:19.604948 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:20.292046 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:20.292005 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:21.038466 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:21.038421 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:21.331640 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:21.331530 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:21.463625 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:21.463565 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:21.833703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:21.833253 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:22.921097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:22.921039 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:23.030257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:23.005618 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:23.066701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:23.052886 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:23.177762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:23.176758 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:23.198153 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:23.198089 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:23.254702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:23.253190 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:23.273122 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:23.273078 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:23.275331 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:23.275297 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:23.466888 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:23.466779 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:23.473805 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:23.473763 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:23.547697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:23.540824 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:23.547697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:23.541263 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:32:23.737761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:23.731238 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Oct 31 03:32:23.922829 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:23.922778 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:24.216703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:24.177854 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:32:24.216703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:24.203288 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:32:24.216703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:24.203356 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:32:24.216703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:24.178327 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:24.224544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:24.224508 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:32:24.233145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:24.233108 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:24.233864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:24.233839 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:32:24.291800 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:24.282053 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:32:24.291800 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:24.285795 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:32:24.309327 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:24.297760 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:24.309327 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:24.298069 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:32:24.309327 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:24.298499 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:32:24.538928 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:24.536403 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:24.585347 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:24.585305 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:32:24.614738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:24.585774 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:32:24.614738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:24.600862 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:32:24.614738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:24.614049 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:32:24.614738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:24.614123 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:32:24.614738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:24.614163 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:32:25.016001 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:25.015935 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:25.320060 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:25.273641 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:25.320060 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:25.275229 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:32:25.320060 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:25.292784 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:32:25.393700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:25.358970 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:25.407741 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:25.395732 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:25.449300 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:25.411136 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:32:25.511243 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:25.509293 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:25.853540 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:25.853478 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:32:25.874298 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:25.874250 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:25.874665 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:25.874621 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:32:25.915702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:25.895579 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:26.275847 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:26.275789 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:26.304668 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:26.302082 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:32:26.315999 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:26.303551 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:26.391747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:26.371580 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:26.401712 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:26.384194 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:26.430027 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:26.413545 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:26.602760 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:26.599230 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:32:26.806341 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:26.806312 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:32:26.874903 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:26.874795 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:26.883170 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:26.883128 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:32:26.922703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:26.921649 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:27.115698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:27.091111 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:27.239703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:27.222641 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded" Oct 31 03:32:27.624573 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:27.618125 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:32:27.643861 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:27.643820 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:32:27.652329 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:27.651330 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:32:27.753842 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:27.753516 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:27.757382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:27.757346 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:32:27.888544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:27.888437 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:28.136442 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:28.136399 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:32:28.170390 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:28.169486 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:32:28.282052 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:28.281991 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:28.324761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:28.322784 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:32:28.333356 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:28.333318 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:28.342039 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:28.342005 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:32:28.345620 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:28.345597 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:32:28.374324 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:28.372995 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:28.399209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:28.392098 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:32:28.407533 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:28.407502 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:32:29.019961 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:29.019913 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:29.646198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:29.646165 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:29.789769 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:29.789300 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:29.920541 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:29.920501 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:30.064009 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:30.063974 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:30.723159 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:30.723099 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:31.059009 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:31.058848 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:31.354647 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:31.354544 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:31.994129 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:31.994091 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:32.288388 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:32.288299 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:32.469315 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:32.469277 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg"] Oct 31 03:32:32.470329 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:32.470289 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" podUID="3b219805-f5a8-4f72-a014-0f629263010a" containerName="maas-api" containerID="cri-o://55da22283ec74ad49dd74a9688cd2a201081963c19c6d7d8b75f077a17f5c0dd" gracePeriod=30 Oct 31 03:32:32.546924 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:32.546840 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:32.696105 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:32.696027 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:32.910448 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:32.896465 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:33.062112 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.062068 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:33.067142 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.067110 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" Oct 31 03:32:33.187716 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.187611 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b219805-f5a8-4f72-a014-0f629263010a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b219805-f5a8-4f72-a014-0f629263010a-metrics-certs\") pod \"3b219805-f5a8-4f72-a014-0f629263010a\" (UID: \"3b219805-f5a8-4f72-a014-0f629263010a\") " Oct 31 03:32:33.187879 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.187741 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b219805-f5a8-4f72-a014-0f629263010a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b219805-f5a8-4f72-a014-0f629263010a-maas-api-tls\") pod \"3b219805-f5a8-4f72-a014-0f629263010a\" (UID: \"3b219805-f5a8-4f72-a014-0f629263010a\") " Oct 31 03:32:33.187926 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.187882 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b219805-f5a8-4f72-a014-0f629263010a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b219805-f5a8-4f72-a014-0f629263010a-openshift-service-ca\") pod \"3b219805-f5a8-4f72-a014-0f629263010a\" (UID: \"3b219805-f5a8-4f72-a014-0f629263010a\") " Oct 31 03:32:33.188159 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.188092 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b219805-f5a8-4f72-a014-0f629263010a-kube-api-access-w8zls\" (UniqueName: \"kubernetes.io/projected/3b219805-f5a8-4f72-a014-0f629263010a-kube-api-access-w8zls\") pod \"3b219805-f5a8-4f72-a014-0f629263010a\" (UID: \"3b219805-f5a8-4f72-a014-0f629263010a\") " Oct 31 03:32:33.188322 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.188295 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b219805-f5a8-4f72-a014-0f629263010a-openshift-service-ca" pod "3b219805-f5a8-4f72-a014-0f629263010a" (UID: "3b219805-f5a8-4f72-a014-0f629263010a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:32:33.188909 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.188881 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b219805-f5a8-4f72-a014-0f629263010a-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:33.189983 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.189958 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b219805-f5a8-4f72-a014-0f629263010a-maas-api-tls" pod "3b219805-f5a8-4f72-a014-0f629263010a" (UID: "3b219805-f5a8-4f72-a014-0f629263010a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:32:33.190164 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.190142 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b219805-f5a8-4f72-a014-0f629263010a-metrics-certs" pod "3b219805-f5a8-4f72-a014-0f629263010a" (UID: "3b219805-f5a8-4f72-a014-0f629263010a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:32:33.190265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.190237 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b219805-f5a8-4f72-a014-0f629263010a-kube-api-access-w8zls" pod "3b219805-f5a8-4f72-a014-0f629263010a" (UID: "3b219805-f5a8-4f72-a014-0f629263010a"). InnerVolumeSpecName "kube-api-access-w8zls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:32:33.291665 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.291624 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8zls\" (UniqueName: \"kubernetes.io/projected/3b219805-f5a8-4f72-a014-0f629263010a-kube-api-access-w8zls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:33.292002 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.291982 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b219805-f5a8-4f72-a014-0f629263010a-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:33.316909 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.316877 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b219805-f5a8-4f72-a014-0f629263010a-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:32:33.344738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.344700 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:33.608577 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.608541 5490 generic.go:356] "Generic (PLEG): container finished" podID="3b219805-f5a8-4f72-a014-0f629263010a" containerID="55da22283ec74ad49dd74a9688cd2a201081963c19c6d7d8b75f077a17f5c0dd" exitCode=0 Oct 31 03:32:33.609102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.608592 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" Oct 31 03:32:33.609102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.608624 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" event={"ID":"3b219805-f5a8-4f72-a014-0f629263010a","Type":"ContainerDied","Data":"55da22283ec74ad49dd74a9688cd2a201081963c19c6d7d8b75f077a17f5c0dd"} Oct 31 03:32:33.609102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.608720 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg" event={"ID":"3b219805-f5a8-4f72-a014-0f629263010a","Type":"ContainerDied","Data":"f3f26c5c495fc9142a1083894b3f4eff70c8380d9c4a3e0c4a113c50449f90b5"} Oct 31 03:32:33.609102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.608727 5490 scope.go:122] "RemoveContainer" containerID="55da22283ec74ad49dd74a9688cd2a201081963c19c6d7d8b75f077a17f5c0dd" Oct 31 03:32:33.620581 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.620553 5490 scope.go:122] "RemoveContainer" containerID="55da22283ec74ad49dd74a9688cd2a201081963c19c6d7d8b75f077a17f5c0dd" Oct 31 03:32:33.620971 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:32:33.620948 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55da22283ec74ad49dd74a9688cd2a201081963c19c6d7d8b75f077a17f5c0dd\": container with ID starting with 55da22283ec74ad49dd74a9688cd2a201081963c19c6d7d8b75f077a17f5c0dd not found: ID does not exist" containerID="55da22283ec74ad49dd74a9688cd2a201081963c19c6d7d8b75f077a17f5c0dd" Oct 31 03:32:33.621060 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.620982 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55da22283ec74ad49dd74a9688cd2a201081963c19c6d7d8b75f077a17f5c0dd"} err="failed to get container status \"55da22283ec74ad49dd74a9688cd2a201081963c19c6d7d8b75f077a17f5c0dd\": rpc error: code = NotFound desc = could not find container \"55da22283ec74ad49dd74a9688cd2a201081963c19c6d7d8b75f077a17f5c0dd\": container with ID starting with 55da22283ec74ad49dd74a9688cd2a201081963c19c6d7d8b75f077a17f5c0dd not found: ID does not exist" Oct 31 03:32:33.627329 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.627297 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg"] Oct 31 03:32:33.634838 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.634802 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1c45857e-5c9479fcb9-rqjsg"] Oct 31 03:32:33.865537 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:33.865410 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:34.174794 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:34.172524 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:34.335719 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:34.315702 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:34.874697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:34.874650 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:35.013955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:35.013914 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:35.578420 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:35.578381 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b219805-f5a8-4f72-a014-0f629263010a" path="/var/lib/kubelet/pods/3b219805-f5a8-4f72-a014-0f629263010a/volumes" Oct 31 03:32:35.876323 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:35.876231 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:36.702003 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:36.701966 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:37.398246 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:37.398195 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:38.386545 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:38.384834 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:38.897349 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:38.897310 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:39.070045 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:39.070009 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:39.421444 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:39.419062 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:40.074626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:40.074591 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:40.216913 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:40.216742 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:41.074025 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:41.073989 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:41.998805 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:41.996998 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:42.761053 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:42.761011 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:43.484625 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:43.484582 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:44.192809 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:44.192770 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:44.879202 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:44.878988 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:45.254271 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:45.254227 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:45.411370 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:45.397304 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:45.597970 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:45.597867 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:46.179525 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:46.179482 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:46.314466 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:46.314428 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:46.465768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:46.464591 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:46.678701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:46.678541 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:47.136660 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:32:47.136625 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:32:47.330187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:47.330149 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:47.487571 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:47.487534 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:47.634901 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:47.634867 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:47.936048 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:47.936012 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:48.247607 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:48.247514 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:49.026522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:49.025826 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:49.607388 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:49.607315 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:50.306207 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:50.306174 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:50.444118 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:50.444015 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:50.975403 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:50.975353 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:51.605055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:51.604898 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:52.308510 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:52.308461 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:52.890235 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:52.890203 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:53.589657 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:53.589613 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:53.761697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:53.758243 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:53.883012 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:53.882916 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:54.282983 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:54.282944 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:54.883249 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:54.883208 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:55.487091 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:55.487049 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:56.342989 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:56.342815 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:57.143706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:57.136726 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:57.564879 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:57.543786 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:58.201748 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:58.201699 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded" Oct 31 03:32:58.277292 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:58.277247 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:58.416761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:58.416668 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:58.728704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:58.727968 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:58.990291 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:58.990196 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:32:59.008966 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:59.008921 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:59.023895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:59.023847 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:59.025453 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:59.025420 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:59.028966 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:59.025407 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:32:59.694549 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:59.694000 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:32:59.694549 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:32:59.694084 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:33:00.284583 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:00.284536 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:00.995320 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:00.995262 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:01.389165 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:01.389075 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:02.195104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:02.194946 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:03.089841 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:03.089792 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:03.652013 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:03.651969 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:03.806702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:03.806487 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:04.020652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:04.020615 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:04.829801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:04.829751 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:05.016830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:05.012938 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:05.187150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:05.186935 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:06.311784 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:06.310238 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:06.904071 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:06.904034 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:07.520995 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:07.520949 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:07.972583 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:07.972515 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:08.227621 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:08.227442 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:08.867160 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:08.867115 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:09.746274 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:09.746233 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:10.170812 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:10.170758 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:10.301469 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:10.301371 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:10.494809 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:10.494718 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:10.804402 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:10.804314 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:11.500452 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:11.500408 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:12.102422 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:12.102378 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:12.775066 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:12.775024 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:13.401163 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:13.401111 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:14.369802 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:14.369752 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:14.995272 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:14.995239 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:15.524971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:15.524926 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:15.657727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:15.656869 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:15.928198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:15.928155 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:16.120547 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:16.120515 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:16.243759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:16.243649 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:16.375911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:16.375871 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:17.102593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:17.102549 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:17.143106 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:33:17.143065 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:33:17.811215 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:17.808695 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:18.384826 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:18.384795 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:18.976931 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:18.976894 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:19.578865 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:19.578804 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:20.184364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:20.184321 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:20.796542 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:20.796504 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:21.305736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:21.305650 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:21.470736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:21.464315 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:21.601141 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:21.601052 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:21.726489 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:21.726452 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:22.511997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:22.511944 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:23.105070 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:23.105029 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:23.241913 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:23.241868 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:23.966165 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:23.966128 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:24.105476 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:24.105432 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:24.445309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:24.445267 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:24.593204 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:24.593170 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:24.955034 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:24.954995 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:25.604172 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:25.604131 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:26.403489 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:26.403456 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:26.980048 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:26.980013 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:27.947034 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:27.946971 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:28.635998 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:28.635952 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:28.777460 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:28.777415 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:28.913629 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:28.913593 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:29.496055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:29.496018 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:29.916737 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:29.916661 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:30.145651 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:30.145614 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:30.487598 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:30.487553 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:31.097043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:31.096991 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:31.778924 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:31.778884 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:32.410590 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:32.410553 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:32.545384 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:32.545343 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:33.090342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:33.090303 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:33.791950 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:33.791898 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:34.502754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:34.502717 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:34.634262 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:34.634218 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:34.766406 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:34.766318 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:35.211270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:35.211234 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:35.895690 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:35.895637 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:36.603300 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:36.603264 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:37.503595 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:37.503549 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:38.122382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:38.122343 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:38.277866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:38.277782 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:38.792782 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:38.792737 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:39.395698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:39.395638 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:40.024890 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:40.024833 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:40.616964 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:40.616791 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:41.270868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:41.270823 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:41.896312 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:41.895506 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:42.783801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:42.783761 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:43.587635 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:43.587593 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:44.298419 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:44.298367 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:44.426182 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:44.426140 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:45.001166 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:45.001135 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:45.576055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:45.576015 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:46.280727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:46.280339 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:46.890209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:46.889904 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:47.129394 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:33:47.129348 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:33:47.485901 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:47.485858 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:48.079224 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:48.079186 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:48.849633 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:48.849598 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:49.543505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:49.543466 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:50.182382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:50.182346 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:50.887510 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:50.887464 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:51.608293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:51.608194 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:52.178255 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:52.178212 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:52.793280 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:52.793242 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:53.459159 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:53.459102 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:53.882688 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:53.882571 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:54.137087 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.136978 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:54.301987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.301948 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:54.664246 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.664210 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j"] Oct 31 03:33:54.665662 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.665457 5490 cpu_manager.go:414] "containerMap: removing container" podUID="bd68eadd-4bc5-4f8d-920f-e6e6babb8141" containerName="istio-proxy" Oct 31 03:33:54.665662 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.665484 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd68eadd-4bc5-4f8d-920f-e6e6babb8141" containerName="istio-proxy" Oct 31 03:33:54.665662 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.665498 5490 cpu_manager.go:414] "containerMap: removing container" podUID="1c52e703-346c-4b26-89b7-a8a8954ad15f" containerName="maas-api" Oct 31 03:33:54.665662 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.665506 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c52e703-346c-4b26-89b7-a8a8954ad15f" containerName="maas-api" Oct 31 03:33:54.665662 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.665525 5490 cpu_manager.go:414] "containerMap: removing container" podUID="3b219805-f5a8-4f72-a014-0f629263010a" containerName="maas-api" Oct 31 03:33:54.665662 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.665532 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b219805-f5a8-4f72-a014-0f629263010a" containerName="maas-api" Oct 31 03:33:54.665662 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.665773 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd68eadd-4bc5-4f8d-920f-e6e6babb8141" containerName="istio-proxy" Oct 31 03:33:54.665662 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.665790 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c52e703-346c-4b26-89b7-a8a8954ad15f" containerName="maas-api" Oct 31 03:33:54.666302 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.665800 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b219805-f5a8-4f72-a014-0f629263010a" containerName="maas-api" Oct 31 03:33:54.670302 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.670280 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.681175 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.681148 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j"] Oct 31 03:33:54.689582 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.689478 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-5396c2-openshift-default-dockercfg-xpxfn\"" Oct 31 03:33:54.741456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.741411 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:54.754868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.754780 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-workload-certs\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.755128 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.754891 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk6jg\" (UniqueName: \"kubernetes.io/projected/dbec9363-80a5-497b-8dd8-45b9fba41328-kube-api-access-bk6jg\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.755128 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.754989 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-podinfo\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.755128 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.755019 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-token\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.755403 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.755173 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-data\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.755403 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.755241 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-credential-socket\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.755518 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.755436 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-workload-socket\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.755610 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.755591 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbec9363-80a5-497b-8dd8-45b9fba41328-istiod-ca-cert\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.755736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.755664 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-envoy\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.871471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.857106 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-data\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.871471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.857222 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-credential-socket\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.871471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.857437 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-workload-socket\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.871471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.857583 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbec9363-80a5-497b-8dd8-45b9fba41328-istiod-ca-cert\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.871471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.857646 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-envoy\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.871471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.857707 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-workload-certs\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.871471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.857737 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk6jg\" (UniqueName: \"kubernetes.io/projected/dbec9363-80a5-497b-8dd8-45b9fba41328-kube-api-access-bk6jg\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.871471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.858211 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-workload-certs\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.871471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.858266 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-podinfo\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.871471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.858295 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-token\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.871471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.858370 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-credential-socket\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.871471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.859055 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-data\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.871471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.859591 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-workload-socket\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.871471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.859598 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbec9363-80a5-497b-8dd8-45b9fba41328-istiod-ca-cert\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.875767 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.873181 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-envoy\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.907700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.904871 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk6jg\" (UniqueName: \"kubernetes.io/projected/dbec9363-80a5-497b-8dd8-45b9fba41328-kube-api-access-bk6jg\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.910249 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.910128 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:54.911124 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.911050 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-podinfo\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.952661 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.950317 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-token\") pod \"e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:54.993296 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:54.992984 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:55.058130 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:55.058096 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:55.329415 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:55.329385 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j"] Oct 31 03:33:55.329415 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:33:55.329397 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbec9363_80a5_497b_8dd8_45b9fba41328.slice/crio-e430b96fd5646df1fcc81003a22f1361cd2fc0f385d15a070971190640671f4e WatchSource:0}: Error finding container e430b96fd5646df1fcc81003a22f1361cd2fc0f385d15a070971190640671f4e: Status 404 returned error can't find the container with id e430b96fd5646df1fcc81003a22f1361cd2fc0f385d15a070971190640671f4e Oct 31 03:33:55.333870 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:55.333834 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:33:55.333991 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:55.333924 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:33:55.333991 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:55.333958 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:33:55.438324 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:55.438280 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" event={"ID":"dbec9363-80a5-497b-8dd8-45b9fba41328","Type":"ContainerStarted","Data":"0f07b92fbded13cb25bc3a9fed826a9aa0bf26995e6498bd4d386f0a0ef0d7c9"} Oct 31 03:33:55.438504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:55.438362 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" event={"ID":"dbec9363-80a5-497b-8dd8-45b9fba41328","Type":"ContainerStarted","Data":"e430b96fd5646df1fcc81003a22f1361cd2fc0f385d15a070971190640671f4e"} Oct 31 03:33:55.468595 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:55.468538 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" podStartSLOduration=1.4685171289999999 podStartE2EDuration="1.468517129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:55.458890511 +0000 UTC m=+1069.993585467" watchObservedRunningTime="2026-10-31 03:33:55.468517129 +0000 UTC m=+1070.003212083" Oct 31 03:33:55.712546 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:55.712501 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:55.994727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:55.994580 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:56.396748 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:56.396589 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:57.000038 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:57.000009 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:57.141988 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:57.141952 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:57.466086 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:57.466050 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:57.467078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:57.467052 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:33:57.779645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:57.778709 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:57.983377 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:57.983343 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:58.130135 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:58.130034 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:58.514236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:58.514198 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:59.093184 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:59.093144 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:33:59.397758 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:33:59.386014 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:00.080207 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:00.080166 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:00.719697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:00.719645 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:01.425543 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:01.425497 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:02.055117 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:02.055034 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:02.669608 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:02.669567 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:03.268525 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:03.268494 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:03.874997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:03.874959 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:04.012328 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:04.012292 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:04.688657 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:04.688612 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:05.341534 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:05.341497 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:05.959494 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:05.959420 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:06.620241 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:06.619960 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:07.288319 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:07.288217 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:07.932213 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:07.930705 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:08.532928 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:08.532873 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:09.167735 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:09.167639 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:09.987742 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:09.987705 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:10.825654 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:10.825616 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:12.434727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:12.390219 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:12.888769 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:12.884798 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:12.895179 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:12.895126 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:13.288709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:13.281978 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:13.326274 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:13.326211 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:13.337104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:13.337052 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:13.436701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:13.428198 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:13.477893 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:13.477829 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:14.164652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:14.164607 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:15.154457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:15.154413 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:16.032371 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:16.032288 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:16.266868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:16.266823 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:16.706607 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:16.706565 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:17.139537 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:34:17.139451 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:34:17.357356 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:17.356648 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:17.961381 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:17.961332 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:18.099321 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:18.097186 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:18.678529 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:18.678488 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:19.284468 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:19.284424 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:19.880360 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:19.880309 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:20.487569 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:20.487513 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:21.102448 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:21.102410 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:21.793842 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:21.793804 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:22.417632 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:22.416515 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:22.991268 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:22.991227 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:23.581344 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:23.581303 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:24.177904 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:24.177861 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:24.806519 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:24.806478 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:25.387367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:25.387314 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:26.035203 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:26.035165 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:26.710099 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:26.710015 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:27.366671 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:27.366621 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:27.995896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:27.995798 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:28.582583 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:28.582511 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:29.215493 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:29.215437 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:29.908438 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:29.908390 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:30.503745 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:30.503618 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:31.184380 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:31.184344 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:31.449610 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:31.449516 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:31.788023 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:31.787913 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:31.963698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:31.961248 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:32.282290 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:32.282241 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:32.428215 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:32.428160 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:32.568459 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:32.568368 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:32.720299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:32.720265 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:32.998050 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:32.998009 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:33.449866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:33.449823 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:33.599214 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:33.599164 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:33.760758 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:33.760210 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:34.484605 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:34.483860 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:35.202204 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:35.202169 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:35.804307 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:35.804270 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:35.954311 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:35.954270 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:36.101833 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:36.101739 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:36.883400 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:36.883366 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:37.100394 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:37.100354 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:37.150099 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:37.150007 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw"] Oct 31 03:34:37.155099 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:37.155071 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" Oct 31 03:34:37.172366 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:37.172335 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw"] Oct 31 03:34:37.175750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:37.175721 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-34cb76\"" Oct 31 03:34:37.222321 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:37.222116 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-34cb76-dockercfg-qbx54\"" Oct 31 03:34:37.353858 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:37.351526 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn78k\" (UniqueName: \"kubernetes.io/projected/42e53881-926d-482d-9557-a86d3ae3f9d9-kube-api-access-zn78k\") pod \"payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw\" (UID: \"42e53881-926d-482d-9557-a86d3ae3f9d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" Oct 31 03:34:37.353858 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:37.351764 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e53881-926d-482d-9557-a86d3ae3f9d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw\" (UID: \"42e53881-926d-482d-9557-a86d3ae3f9d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" Oct 31 03:34:37.442371 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:37.442330 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:37.453459 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:37.453378 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e53881-926d-482d-9557-a86d3ae3f9d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw\" (UID: \"42e53881-926d-482d-9557-a86d3ae3f9d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" Oct 31 03:34:37.453694 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:37.453659 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn78k\" (UniqueName: \"kubernetes.io/projected/42e53881-926d-482d-9557-a86d3ae3f9d9-kube-api-access-zn78k\") pod \"payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw\" (UID: \"42e53881-926d-482d-9557-a86d3ae3f9d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" Oct 31 03:34:37.454693 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:37.454653 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e53881-926d-482d-9557-a86d3ae3f9d9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw\" (UID: \"42e53881-926d-482d-9557-a86d3ae3f9d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" Oct 31 03:34:37.473892 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:37.473850 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn78k\" (UniqueName: \"kubernetes.io/projected/42e53881-926d-482d-9557-a86d3ae3f9d9-kube-api-access-zn78k\") pod \"payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw\" (UID: \"42e53881-926d-482d-9557-a86d3ae3f9d9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" Oct 31 03:34:37.770944 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:37.770842 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" Oct 31 03:34:37.892142 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:37.892015 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:38.058514 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:38.057990 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw"] Oct 31 03:34:38.059271 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:34:38.059215 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e53881_926d_482d_9557_a86d3ae3f9d9.slice/crio-f0cda0ed41511dbe873234782faa3da6d83f41e4029bfd84e48a591cd3f6c2b6 WatchSource:0}: Error finding container f0cda0ed41511dbe873234782faa3da6d83f41e4029bfd84e48a591cd3f6c2b6: Status 404 returned error can't find the container with id f0cda0ed41511dbe873234782faa3da6d83f41e4029bfd84e48a591cd3f6c2b6 Oct 31 03:34:38.149527 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:38.149487 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" event={"ID":"42e53881-926d-482d-9557-a86d3ae3f9d9","Type":"ContainerStarted","Data":"dfe524b27adecb16940953181c7b9e812fd05a362305d00c551e3c6d3d67a10b"} Oct 31 03:34:38.149744 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:38.149553 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" event={"ID":"42e53881-926d-482d-9557-a86d3ae3f9d9","Type":"ContainerStarted","Data":"f0cda0ed41511dbe873234782faa3da6d83f41e4029bfd84e48a591cd3f6c2b6"} Oct 31 03:34:38.149744 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:38.149592 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" Oct 31 03:34:38.172654 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:38.172595 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" podStartSLOduration=1.172580414 podStartE2EDuration="1.172580414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:38.168388964 +0000 UTC m=+1112.703083920" watchObservedRunningTime="2026-10-31 03:34:38.172580414 +0000 UTC m=+1112.707275412" Oct 31 03:34:38.759832 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:38.759792 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:39.399386 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:39.399348 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:39.995775 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:39.990951 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:40.579805 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:40.579765 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:41.243044 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:41.243004 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:41.915258 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:41.915196 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:42.506990 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:42.506343 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:43.087762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:43.087723 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl"] Oct 31 03:34:43.093721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:43.093692 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" Oct 31 03:34:43.101989 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:43.101956 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-34cb76-serving-cert\"" Oct 31 03:34:43.107254 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:43.107216 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl"] Oct 31 03:34:43.117269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:43.116158 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:43.155810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:43.155753 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c026b4-65c7-4d57-9945-d4ebfb66ca70-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl\" (UID: \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" Oct 31 03:34:43.156115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:43.156094 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-842q5\" (UniqueName: \"kubernetes.io/projected/76c026b4-65c7-4d57-9945-d4ebfb66ca70-kube-api-access-842q5\") pod \"maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl\" (UID: \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" Oct 31 03:34:43.156207 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:43.156143 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c026b4-65c7-4d57-9945-d4ebfb66ca70-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl\" (UID: \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" Oct 31 03:34:43.156298 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:43.156280 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76c026b4-65c7-4d57-9945-d4ebfb66ca70-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl\" (UID: \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" Oct 31 03:34:43.248922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:43.248869 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:43.257774 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:43.257732 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-842q5\" (UniqueName: \"kubernetes.io/projected/76c026b4-65c7-4d57-9945-d4ebfb66ca70-kube-api-access-842q5\") pod \"maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl\" (UID: \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" Oct 31 03:34:43.258046 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:43.257781 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c026b4-65c7-4d57-9945-d4ebfb66ca70-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl\" (UID: \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" Oct 31 03:34:43.258046 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:43.257852 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76c026b4-65c7-4d57-9945-d4ebfb66ca70-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl\" (UID: \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" Oct 31 03:34:43.258046 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:43.257956 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c026b4-65c7-4d57-9945-d4ebfb66ca70-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl\" (UID: \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" Oct 31 03:34:43.258251 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:34:43.258056 5490 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-serving-cert: secret "maas-api-e2e-shared-a-w5-34cb76-serving-cert" not found Oct 31 03:34:43.258251 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:34:43.258145 5490 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76c026b4-65c7-4d57-9945-d4ebfb66ca70-maas-api-tls podName:76c026b4-65c7-4d57-9945-d4ebfb66ca70 nodeName:}" failed. No retries permitted until 2026-10-31 03:34:43.7581216 +0000 UTC m=+1118.292816553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/76c026b4-65c7-4d57-9945-d4ebfb66ca70-maas-api-tls") pod "maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" (UID: "76c026b4-65c7-4d57-9945-d4ebfb66ca70") : secret "maas-api-e2e-shared-a-w5-34cb76-serving-cert" not found Oct 31 03:34:43.258642 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:43.258618 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c026b4-65c7-4d57-9945-d4ebfb66ca70-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl\" (UID: \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" Oct 31 03:34:43.260775 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:43.260753 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c026b4-65c7-4d57-9945-d4ebfb66ca70-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl\" (UID: \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" Oct 31 03:34:43.275760 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:43.267652 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-842q5\" (UniqueName: \"kubernetes.io/projected/76c026b4-65c7-4d57-9945-d4ebfb66ca70-kube-api-access-842q5\") pod \"maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl\" (UID: \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" Oct 31 03:34:43.680970 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:43.680927 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:43.768008 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:43.767959 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76c026b4-65c7-4d57-9945-d4ebfb66ca70-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl\" (UID: \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" Oct 31 03:34:43.784335 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:43.784297 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76c026b4-65c7-4d57-9945-d4ebfb66ca70-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl\" (UID: \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" Oct 31 03:34:44.001135 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:44.001038 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:44.023117 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:44.023087 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" Oct 31 03:34:44.353335 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:44.353302 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl"] Oct 31 03:34:44.354083 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:34:44.354050 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c026b4_65c7_4d57_9945_d4ebfb66ca70.slice/crio-e2d087b176d7c27a81801a0a685d26800cdfc9d1b39b61b4c4751b7fd36c7915 WatchSource:0}: Error finding container e2d087b176d7c27a81801a0a685d26800cdfc9d1b39b61b4c4751b7fd36c7915: Status 404 returned error can't find the container with id e2d087b176d7c27a81801a0a685d26800cdfc9d1b39b61b4c4751b7fd36c7915 Oct 31 03:34:44.592092 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:44.592054 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:45.214765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:45.214724 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" event={"ID":"76c026b4-65c7-4d57-9945-d4ebfb66ca70","Type":"ContainerStarted","Data":"b785a251429aa9dc9811da6efd751dd1824b4436e966c96790aa9465afabad00"} Oct 31 03:34:45.214972 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:45.214812 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" event={"ID":"76c026b4-65c7-4d57-9945-d4ebfb66ca70","Type":"ContainerStarted","Data":"e2d087b176d7c27a81801a0a685d26800cdfc9d1b39b61b4c4751b7fd36c7915"} Oct 31 03:34:45.215940 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:45.215912 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" Oct 31 03:34:45.283633 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:45.283562 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" podStartSLOduration=1.835065862 podStartE2EDuration="2.272666628s" totalImagesPullingTime="437.600766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:45.269123199 +0000 UTC m=+1119.803818155" watchObservedRunningTime="2026-10-31 03:34:45.272666628 +0000 UTC m=+1119.807361584" Oct 31 03:34:45.299587 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:45.299548 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:45.432538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:45.432495 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:45.926904 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:45.926864 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:46.514762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:46.514720 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:47.133126 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:34:47.133083 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:34:47.229594 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:47.229323 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:48.025518 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:48.025474 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:48.648808 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:48.648771 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:49.161700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:49.161424 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" Oct 31 03:34:49.245867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:49.245739 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:49.416905 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:49.414706 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:49.998899 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:49.998854 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:50.815878 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:50.815829 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:51.224866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:51.224773 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:51.770700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:51.764793 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:52.315030 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:52.314994 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" Oct 31 03:34:52.726054 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:52.702571 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:52.893702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:52.883622 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:52.916605 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:52.903116 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:52.916605 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:52.914793 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:53.190477 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:53.190437 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:53.281823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:53.281764 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:53.364702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:53.334775 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:34:53.364702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:53.335110 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:34:53.481762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:53.480763 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:53.498702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:53.482545 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:34:53.518947 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:53.518901 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:34:53.715707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:53.714755 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:53.743949 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:53.743812 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:53.760702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:53.756798 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:34:53.966462 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:53.917560 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:53.966462 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:53.919652 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:34:53.968025 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:53.967980 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:34:54.022245 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:53.976219 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:34:54.022245 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:53.971014 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:54.092701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.092322 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:34:54.419994 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.411121 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:34:54.427404 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.427366 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:34:54.694112 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.448546 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:34:54.696086 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.448582 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:54.696086 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.453551 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": context deadline exceeded" Oct 31 03:34:54.696086 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.489451 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:54.696086 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.598493 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:54.699186 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.697970 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:34:54.699186 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.699019 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:34:54.699379 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.699320 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:34:54.699379 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.699368 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:34:54.699487 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.699401 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:34:54.699487 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.699417 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:34:54.701234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.701214 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:34:54.701738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.701510 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:54.701738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.701634 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:34:54.702922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.702368 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:34:54.728186 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.728155 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:34:54.805468 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.805433 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:34:54.827156 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.827125 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:34:54.888294 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:54.886512 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:34:55.377503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:55.377407 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:56.110943 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:56.102394 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:56.946198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:56.946158 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:57.573458 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:57.573417 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:58.216149 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:58.216108 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:58.514926 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:58.514829 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:34:58.514926 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:58.514894 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" podUID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:34:58.874835 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:58.874709 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:34:59.604807 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:34:59.604670 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:00.185192 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:00.185137 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:00.793476 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:00.793435 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:01.394289 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:01.394120 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:02.352642 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:02.352600 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:03.169096 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:03.169060 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:03.221496 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:03.221403 5490 generic.go:356] "Generic (PLEG): container finished" podID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerID="0f07b92fbded13cb25bc3a9fed826a9aa0bf26995e6498bd4d386f0a0ef0d7c9" exitCode=0 Oct 31 03:35:03.221655 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:03.221520 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" event={"ID":"dbec9363-80a5-497b-8dd8-45b9fba41328","Type":"ContainerDied","Data":"0f07b92fbded13cb25bc3a9fed826a9aa0bf26995e6498bd4d386f0a0ef0d7c9"} Oct 31 03:35:03.222227 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:03.222197 5490 scope.go:122] "RemoveContainer" containerID="0f07b92fbded13cb25bc3a9fed826a9aa0bf26995e6498bd4d386f0a0ef0d7c9" Oct 31 03:35:03.224341 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:03.224309 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:35:03.224465 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:03.224448 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:35:03.224534 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:03.224513 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:35:04.256314 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:04.256269 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" event={"ID":"dbec9363-80a5-497b-8dd8-45b9fba41328","Type":"ContainerStarted","Data":"424280cd883a00cc9ef9cb568a8001d97a878f7a2f6c55f4a3e02f39e1fe4787"} Oct 31 03:35:04.994021 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:04.993963 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:35:04.994274 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:04.994250 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 03:35:04.994368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:04.994320 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" podUID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 03:35:06.005915 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:06.005802 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 03:35:06.006421 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:06.005955 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" podUID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 03:35:06.920328 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:06.920289 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq"] Oct 31 03:35:06.925301 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:06.925265 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:06.960174 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:06.960141 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-ab21e4-openshift-default-dockercfg-mdrmx\"" Oct 31 03:35:06.976121 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:06.975136 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq"] Oct 31 03:35:06.994833 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:06.994795 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 03:35:06.995027 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:06.994863 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" podUID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 03:35:07.173759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.173037 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-workload-socket\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.173759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.173098 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-data\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.173759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.173134 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-envoy\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.173759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.173215 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istiod-ca-cert\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.173759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.173274 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-workload-certs\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.173759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.173396 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-token\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.173759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.173531 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-podinfo\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.173759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.173598 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-credential-socket\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.173759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.173629 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk7f5\" (UniqueName: \"kubernetes.io/projected/a8a2a69d-e4b7-4338-a7b4-22042b94432f-kube-api-access-vk7f5\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.411850 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.411798 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-token\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.412065 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.411979 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-podinfo\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.412065 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.412056 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-credential-socket\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.412193 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.412091 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk7f5\" (UniqueName: \"kubernetes.io/projected/a8a2a69d-e4b7-4338-a7b4-22042b94432f-kube-api-access-vk7f5\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.412193 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.412146 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-workload-socket\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.412193 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.412178 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-data\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.412357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.412212 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-envoy\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.412357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.412302 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istiod-ca-cert\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.412357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.412350 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-workload-certs\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.412823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.412780 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-workload-certs\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.413125 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.413089 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-workload-socket\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.415251 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.415224 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istiod-ca-cert\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.415522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.415504 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-credential-socket\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.415980 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.415957 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-podinfo\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.416227 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.416205 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-data\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.417017 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.416982 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-envoy\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.423511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.423490 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-token\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.423728 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.423669 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk7f5\" (UniqueName: \"kubernetes.io/projected/a8a2a69d-e4b7-4338-a7b4-22042b94432f-kube-api-access-vk7f5\") pod \"e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.584335 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.584249 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:07.994805 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.994770 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 03:35:07.994982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:07.994827 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" podUID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 03:35:08.181036 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:08.181000 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq"] Oct 31 03:35:08.181594 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:35:08.181566 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a2a69d_e4b7_4338_a7b4_22042b94432f.slice/crio-0562dfe171ae30048b707d66b745f588cd3729f4a6d22e7c45fad7d57f276e71 WatchSource:0}: Error finding container 0562dfe171ae30048b707d66b745f588cd3729f4a6d22e7c45fad7d57f276e71: Status 404 returned error can't find the container with id 0562dfe171ae30048b707d66b745f588cd3729f4a6d22e7c45fad7d57f276e71 Oct 31 03:35:08.184555 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:08.184526 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:35:08.184631 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:08.184612 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:35:08.184669 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:08.184643 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:35:08.400232 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:08.400188 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" event={"ID":"a8a2a69d-e4b7-4338-a7b4-22042b94432f","Type":"ContainerStarted","Data":"b1fd0c247d873205c8e76e9c6e1b1c75b4d4095c36cc0b09c203f58771e7431c"} Oct 31 03:35:08.400407 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:08.400246 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" event={"ID":"a8a2a69d-e4b7-4338-a7b4-22042b94432f","Type":"ContainerStarted","Data":"0562dfe171ae30048b707d66b745f588cd3729f4a6d22e7c45fad7d57f276e71"} Oct 31 03:35:08.422892 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:08.422840 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" podStartSLOduration=2.422822478 podStartE2EDuration="2.422822478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:08.421430523 +0000 UTC m=+1142.956125479" watchObservedRunningTime="2026-10-31 03:35:08.422822478 +0000 UTC m=+1142.957517433" Oct 31 03:35:08.598759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:08.598195 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:09.000714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:08.997355 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 31 03:35:09.000714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:08.997433 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" podUID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 31 03:35:09.375909 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:09.375812 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:09.508737 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:09.508700 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:09.592901 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:09.592870 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:10.003028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:10.002994 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:35:10.005929 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:10.004884 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:35:10.005929 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:10.005668 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:35:10.295510 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:10.295417 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:10.423752 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:10.423717 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:10.424703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:10.424669 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:35:10.778293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:10.778250 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:10.909017 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:10.908987 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:11.080211 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:11.080086 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:12.013540 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:12.013495 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:12.159969 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:12.159935 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:12.336806 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:12.322107 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:12.467312 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:12.466693 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:13.283172 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:13.283133 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:13.619746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:13.619625 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:13.763469 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:13.763428 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:14.296115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:14.296067 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:14.947706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:14.947461 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:15.377754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:15.377455 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:15.513338 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:15.513301 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:15.913395 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:15.913351 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:16.334092 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:16.333985 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:16.615589 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:16.615486 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:17.130563 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:35:17.130520 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:35:17.262626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:17.262580 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:17.883757 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:17.883719 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:18.576282 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:18.576242 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:19.181725 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.181670 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74"] Oct 31 03:35:19.187203 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.187175 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" Oct 31 03:35:19.202600 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.202569 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74"] Oct 31 03:35:19.203533 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.203487 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-ab21e4-serving-cert\"" Oct 31 03:35:19.265421 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.265385 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74\" (UID: \"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" Oct 31 03:35:19.265597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.265430 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74\" (UID: \"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" Oct 31 03:35:19.265597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.265568 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74\" (UID: \"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" Oct 31 03:35:19.265748 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.265655 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x4mm\" (UniqueName: \"kubernetes.io/projected/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-kube-api-access-7x4mm\") pod \"maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74\" (UID: \"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" Oct 31 03:35:19.281310 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.281262 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:19.367230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.367183 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74\" (UID: \"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" Oct 31 03:35:19.367546 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.367526 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x4mm\" (UniqueName: \"kubernetes.io/projected/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-kube-api-access-7x4mm\") pod \"maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74\" (UID: \"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" Oct 31 03:35:19.368427 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.368395 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74\" (UID: \"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" Oct 31 03:35:19.368607 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.368588 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74\" (UID: \"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" Oct 31 03:35:19.371173 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.371088 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74\" (UID: \"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" Oct 31 03:35:19.382478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.382417 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74\" (UID: \"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" Oct 31 03:35:19.408710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.398395 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74\" (UID: \"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" Oct 31 03:35:19.431720 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.431433 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:19.434466 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.433284 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x4mm\" (UniqueName: \"kubernetes.io/projected/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-kube-api-access-7x4mm\") pod \"maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74\" (UID: \"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" Oct 31 03:35:19.519462 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.517125 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" Oct 31 03:35:19.615624 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.615586 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:19.869048 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:19.866145 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74"] Oct 31 03:35:19.874155 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:35:19.873329 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c359bb_eb6c_4fc6_8a49_86e4d4ca43ff.slice/crio-776dea0842237cc39ee362f893edbac0a082fca3837d3037c6aefedcf4525f3c WatchSource:0}: Error finding container 776dea0842237cc39ee362f893edbac0a082fca3837d3037c6aefedcf4525f3c: Status 404 returned error can't find the container with id 776dea0842237cc39ee362f893edbac0a082fca3837d3037c6aefedcf4525f3c Oct 31 03:35:20.041298 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.041251 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8"] Oct 31 03:35:20.047422 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.047391 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.071212 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.071159 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-d4145b-openshift-default-dockercfg-ldpr2\"" Oct 31 03:35:20.075652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.075612 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8"] Oct 31 03:35:20.084626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.083921 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-data\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.084626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.083997 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-workload-certs\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.084626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.084081 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85mjd\" (UniqueName: \"kubernetes.io/projected/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-kube-api-access-85mjd\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.084626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.084184 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istiod-ca-cert\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.084626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.084248 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-workload-socket\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.084626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.084324 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-token\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.084626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.084378 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-podinfo\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.084626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.084498 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-credential-socket\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.084626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.084539 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-envoy\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.121564 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.121424 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:20.225707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.217215 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-data\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.225707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.217306 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-workload-certs\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.225707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.217387 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85mjd\" (UniqueName: \"kubernetes.io/projected/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-kube-api-access-85mjd\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.225707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.217482 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istiod-ca-cert\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.225707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.217546 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-workload-socket\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.225707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.217623 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-token\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.225707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.222729 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-podinfo\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.225707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.222918 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-credential-socket\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.225707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.222973 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-envoy\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.225707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.224297 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istiod-ca-cert\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.225707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.224555 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-data\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.225707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.224810 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-workload-certs\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.225707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.225588 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-workload-socket\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.233065 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.233029 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-credential-socket\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.284797 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.260185 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:20.287391 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.287358 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-podinfo\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.341722 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.297521 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85mjd\" (UniqueName: \"kubernetes.io/projected/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-kube-api-access-85mjd\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.341722 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.318367 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-token\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.366709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.360376 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-envoy\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.404764 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.395725 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:20.552434 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.552355 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" event={"ID":"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff","Type":"ContainerStarted","Data":"776dea0842237cc39ee362f893edbac0a082fca3837d3037c6aefedcf4525f3c"} Oct 31 03:35:20.781393 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.781283 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:20.804791 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.804192 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8"] Oct 31 03:35:20.805561 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:35:20.805452 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91c38ef_6298_4315_9a8f_bedd67fcdd1c.slice/crio-c3e01c3c583bca436c46e3783b30492a4af398611d0f66b369de6584db1d68b6 WatchSource:0}: Error finding container c3e01c3c583bca436c46e3783b30492a4af398611d0f66b369de6584db1d68b6: Status 404 returned error can't find the container with id c3e01c3c583bca436c46e3783b30492a4af398611d0f66b369de6584db1d68b6 Oct 31 03:35:20.818828 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.818784 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:35:20.818941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.818882 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:35:20.818941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:20.818921 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:35:21.013359 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:21.013319 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:21.563323 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:21.563282 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" event={"ID":"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff","Type":"ContainerStarted","Data":"e3b25ee8fc97e7bbd6f93e95feb4abe217e500bc811341e1056c60ab3e0675cd"} Oct 31 03:35:21.564080 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:21.564050 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" Oct 31 03:35:21.566580 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:21.566551 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" event={"ID":"f91c38ef-6298-4315-9a8f-bedd67fcdd1c","Type":"ContainerStarted","Data":"13b21d79d292a339aad82c96083618172b35ee0058c235ccc96c8e15faa8a955"} Oct 31 03:35:21.566808 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:21.566606 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" event={"ID":"f91c38ef-6298-4315-9a8f-bedd67fcdd1c","Type":"ContainerStarted","Data":"c3e01c3c583bca436c46e3783b30492a4af398611d0f66b369de6584db1d68b6"} Oct 31 03:35:21.585282 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:21.585231 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" podStartSLOduration=1.934030467 podStartE2EDuration="2.58521725s" totalImagesPullingTime="651.186783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:21.581563428 +0000 UTC m=+1156.116258384" watchObservedRunningTime="2026-10-31 03:35:21.58521725 +0000 UTC m=+1156.119912204" Oct 31 03:35:21.598125 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:21.598090 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:21.606239 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:21.606185 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" podStartSLOduration=1.606170608 podStartE2EDuration="1.606170608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:21.601040983 +0000 UTC m=+1156.135735939" watchObservedRunningTime="2026-10-31 03:35:21.606170608 +0000 UTC m=+1156.140865607" Oct 31 03:35:21.756706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:21.756283 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:21.894410 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:21.894318 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:22.399121 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:22.399087 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:22.406607 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:22.406575 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:22.489709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:22.489658 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:22.577122 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:22.577081 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:22.577980 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:22.577959 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:35:23.434827 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:23.434781 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:24.061797 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:24.061765 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:24.216702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:24.208577 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:25.000930 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:25.000894 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:25.411432 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:25.411379 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:25.547727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:25.547685 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:26.255757 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:26.255662 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:27.079735 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:27.078739 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:27.889386 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:27.889342 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:28.599557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:28.599505 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" Oct 31 03:35:29.248699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:29.248512 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:29.720847 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:29.720799 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:35:29.814860 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:29.814816 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:30.406041 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:30.405991 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 31 03:35:30.478946 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:30.478886 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:35:30.761766 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:30.736052 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:35:30.761766 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:30.746747 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:35:30.761766 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:30.747815 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:35:30.834701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:30.817866 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:35:30.868175 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:30.867765 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:35:30.904374 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:30.904336 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:30.918353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:30.918288 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:35:31.749011 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:31.748979 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:32.430076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:32.430043 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:32.558855 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:32.558817 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:32.800841 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:32.800547 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:32.963044 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:32.963005 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:33.793308 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:33.793265 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:34.522701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:34.521859 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:35.005029 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:35.004996 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:35.405924 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:35.405882 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:35.704509 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:35.704407 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:35.988879 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:35.988782 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:36.396594 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:36.396506 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:37.085290 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:37.085248 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:37.236129 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:37.236081 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:37.429065 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:37.429006 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:37.610710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:37.610656 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:38.005628 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:38.005589 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:38.438187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:38.438147 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:38.583970 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:38.583933 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:38.729291 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:38.729183 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:39.106564 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:39.106457 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:39.678725 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:39.678121 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:40.370423 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:40.370366 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:40.568779 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:40.568665 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:41.075582 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:41.075220 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:41.252702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:41.220599 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:41.745355 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:41.745321 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:41.899012 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:41.898829 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:42.812657 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:42.812607 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:43.430203 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:43.429779 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:44.210171 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:44.210122 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:44.901995 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:44.901955 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:45.120113 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:45.119484 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:45.803762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:45.803672 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:45.952109 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:45.950419 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:46.576808 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:46.576763 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:47.130112 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:35:47.130077 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:35:47.286226 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:47.286170 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:47.974154 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:47.974117 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:48.335049 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:48.334952 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:48.661763 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:48.652262 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:48.837578 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:48.837528 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:49.167226 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:49.167190 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:49.503687 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:49.503582 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:49.643215 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:49.643179 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:50.034965 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:50.034904 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:50.378115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:50.378022 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:50.933971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:50.933926 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:51.095486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:51.095449 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:51.814856 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:51.814818 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:52.400924 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:52.400885 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:53.099034 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:53.098997 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:53.709237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:53.707961 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:53.860542 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:53.860507 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:54.690301 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:54.690261 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:55.055211 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:55.055118 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:55.075494 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:55.075458 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd"] Oct 31 03:35:55.089135 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:55.089109 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" Oct 31 03:35:55.094656 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:55.094628 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd"] Oct 31 03:35:55.094933 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:55.094905 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-86a834\"" Oct 31 03:35:55.095055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:55.094978 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-86a834-dockercfg-tvqfk\"" Oct 31 03:35:55.247706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:55.243878 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw7jp\" (UniqueName: \"kubernetes.io/projected/bae51678-b17d-4a9c-98da-77d3ecf0d6ed-kube-api-access-lw7jp\") pod \"payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd\" (UID: \"bae51678-b17d-4a9c-98da-77d3ecf0d6ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" Oct 31 03:35:55.247706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:55.244225 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae51678-b17d-4a9c-98da-77d3ecf0d6ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd\" (UID: \"bae51678-b17d-4a9c-98da-77d3ecf0d6ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" Oct 31 03:35:55.322779 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:55.320253 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:55.360726 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:55.360167 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae51678-b17d-4a9c-98da-77d3ecf0d6ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd\" (UID: \"bae51678-b17d-4a9c-98da-77d3ecf0d6ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" Oct 31 03:35:55.360726 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:55.360314 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw7jp\" (UniqueName: \"kubernetes.io/projected/bae51678-b17d-4a9c-98da-77d3ecf0d6ed-kube-api-access-lw7jp\") pod \"payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd\" (UID: \"bae51678-b17d-4a9c-98da-77d3ecf0d6ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" Oct 31 03:35:55.364715 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:55.361750 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae51678-b17d-4a9c-98da-77d3ecf0d6ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd\" (UID: \"bae51678-b17d-4a9c-98da-77d3ecf0d6ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" Oct 31 03:35:55.392252 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:55.379592 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw7jp\" (UniqueName: \"kubernetes.io/projected/bae51678-b17d-4a9c-98da-77d3ecf0d6ed-kube-api-access-lw7jp\") pod \"payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd\" (UID: \"bae51678-b17d-4a9c-98da-77d3ecf0d6ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" Oct 31 03:35:55.419774 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:55.417783 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" Oct 31 03:35:55.749729 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:55.747586 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd"] Oct 31 03:35:55.752395 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:35:55.752353 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae51678_b17d_4a9c_98da_77d3ecf0d6ed.slice/crio-3ec677065186efa154227bf21d684683f8dae74dc91b0f47c89d27006427cad9 WatchSource:0}: Error finding container 3ec677065186efa154227bf21d684683f8dae74dc91b0f47c89d27006427cad9: Status 404 returned error can't find the container with id 3ec677065186efa154227bf21d684683f8dae74dc91b0f47c89d27006427cad9 Oct 31 03:35:55.796231 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:55.796197 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:55.851398 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:55.851355 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" event={"ID":"bae51678-b17d-4a9c-98da-77d3ecf0d6ed","Type":"ContainerStarted","Data":"3ec677065186efa154227bf21d684683f8dae74dc91b0f47c89d27006427cad9"} Oct 31 03:35:56.475624 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:56.475498 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:56.863605 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:56.863496 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" event={"ID":"bae51678-b17d-4a9c-98da-77d3ecf0d6ed","Type":"ContainerStarted","Data":"bbacbd77c0b5d8febb4f46075637f1864b7a1d701c8274afde83b6e0d191211f"} Oct 31 03:35:56.863840 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:56.863620 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" Oct 31 03:35:56.884184 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:56.884119 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" podStartSLOduration=1.8841046769999998 podStartE2EDuration="1.884104677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:56.881076472 +0000 UTC m=+1191.415771429" watchObservedRunningTime="2026-10-31 03:35:56.884104677 +0000 UTC m=+1191.418799632" Oct 31 03:35:57.361391 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:57.361349 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:57.836186 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:57.836153 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx"] Oct 31 03:35:57.840864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:57.840844 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:57.847401 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:57.847368 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-50d2d14c-openshift-default-dockercfg-vtnzt\"" Oct 31 03:35:57.859089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:57.859059 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx"] Oct 31 03:35:57.923696 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:57.923489 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50589d85-59be-4355-98ce-d7fd4cd6d331-istiod-ca-cert\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:57.923696 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:57.923607 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-workload-certs\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:57.923696 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:57.923660 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-data\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:57.924087 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:57.923742 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-podinfo\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:57.924087 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:57.923805 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-token\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:57.924087 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:57.923844 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-credential-socket\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:57.924087 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:57.923943 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmt29\" (UniqueName: \"kubernetes.io/projected/50589d85-59be-4355-98ce-d7fd4cd6d331-kube-api-access-tmt29\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:57.924087 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:57.924038 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-workload-socket\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:57.924087 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:57.924073 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-envoy\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:58.026138 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.026096 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-token\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:58.026361 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.026150 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-credential-socket\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:58.026361 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.026232 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmt29\" (UniqueName: \"kubernetes.io/projected/50589d85-59be-4355-98ce-d7fd4cd6d331-kube-api-access-tmt29\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:58.026361 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.026315 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-workload-socket\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:58.026361 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.026344 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-envoy\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:58.026691 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.026513 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50589d85-59be-4355-98ce-d7fd4cd6d331-istiod-ca-cert\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:58.026691 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.026627 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-workload-certs\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:58.026691 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.026629 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-credential-socket\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:58.026691 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.026670 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-data\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:58.026905 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.026702 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-workload-socket\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:58.026905 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.026725 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-podinfo\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:58.027024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.026942 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-workload-certs\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:58.027024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.026992 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-data\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:58.027361 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.027332 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50589d85-59be-4355-98ce-d7fd4cd6d331-istiod-ca-cert\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:58.028880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.028853 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-envoy\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:58.029311 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.029289 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-podinfo\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:58.034545 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.034518 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-token\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:58.034650 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.034597 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmt29\" (UniqueName: \"kubernetes.io/projected/50589d85-59be-4355-98ce-d7fd4cd6d331-kube-api-access-tmt29\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:58.173529 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.173496 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:58.189573 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.189502 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:58.512697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.512654 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx"] Oct 31 03:35:58.513845 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:35:58.513812 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50589d85_59be_4355_98ce_d7fd4cd6d331.slice/crio-ce81f2e75a7ad45ffb1ad69e4ca14b931baa5f012f9c4686721db46d9aa7fd3f WatchSource:0}: Error finding container ce81f2e75a7ad45ffb1ad69e4ca14b931baa5f012f9c4686721db46d9aa7fd3f: Status 404 returned error can't find the container with id ce81f2e75a7ad45ffb1ad69e4ca14b931baa5f012f9c4686721db46d9aa7fd3f Oct 31 03:35:58.517033 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.516998 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:35:58.517128 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.517068 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:35:58.517128 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.517097 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:35:58.780358 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.780239 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:58.896520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.896482 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" event={"ID":"50589d85-59be-4355-98ce-d7fd4cd6d331","Type":"ContainerStarted","Data":"d771073eec5f57ef43e166193854180fdc1704f1dd9b1675384528c754d51c4c"} Oct 31 03:35:58.896988 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.896548 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" event={"ID":"50589d85-59be-4355-98ce-d7fd4cd6d331","Type":"ContainerStarted","Data":"ce81f2e75a7ad45ffb1ad69e4ca14b931baa5f012f9c4686721db46d9aa7fd3f"} Oct 31 03:35:58.920132 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:58.920069 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" podStartSLOduration=1.920051457 podStartE2EDuration="1.920051457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:58.919805527 +0000 UTC m=+1193.454500475" watchObservedRunningTime="2026-10-31 03:35:58.920051457 +0000 UTC m=+1193.454746413" Oct 31 03:35:59.174065 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:59.174015 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:35:59.175845 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:59.175817 5490 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 31 03:35:59.175985 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:59.175867 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" podUID="50589d85-59be-4355-98ce-d7fd4cd6d331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 31 03:35:59.374691 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:59.374648 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:35:59.993399 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:35:59.993291 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:00.174599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:00.174568 5490 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 31 03:36:00.174813 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:00.174616 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" podUID="50589d85-59be-4355-98ce-d7fd4cd6d331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 31 03:36:00.718340 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:00.718311 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:01.174210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:01.174169 5490 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 31 03:36:01.174616 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:01.174242 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" podUID="50589d85-59be-4355-98ce-d7fd4cd6d331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 31 03:36:01.485756 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:01.485300 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:02.174299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:02.174260 5490 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 31 03:36:02.174799 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:02.174332 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" podUID="50589d85-59be-4355-98ce-d7fd4cd6d331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 31 03:36:02.212409 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:02.212365 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:02.989785 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:02.989743 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:03.174213 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:03.174174 5490 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 31 03:36:03.174398 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:03.174265 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" podUID="50589d85-59be-4355-98ce-d7fd4cd6d331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 31 03:36:03.808733 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:03.808592 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:04.173964 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:04.173928 5490 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 31 03:36:04.174253 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:04.174227 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" podUID="50589d85-59be-4355-98ce-d7fd4cd6d331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 31 03:36:04.932871 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:04.932832 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:05.247759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.183823 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:05.247759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.185968 5490 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 31 03:36:05.247759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.186032 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" podUID="50589d85-59be-4355-98ce-d7fd4cd6d331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 31 03:36:05.345258 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.345222 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg"] Oct 31 03:36:05.382274 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.382240 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" Oct 31 03:36:05.389931 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.389895 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg"] Oct 31 03:36:05.394223 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.392280 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-86a834-serving-cert\"" Oct 31 03:36:05.482711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.474224 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8353165-d9b4-445a-9a11-5fdf04307860-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg\" (UID: \"b8353165-d9b4-445a-9a11-5fdf04307860\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" Oct 31 03:36:05.483041 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.483013 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrbw7\" (UniqueName: \"kubernetes.io/projected/b8353165-d9b4-445a-9a11-5fdf04307860-kube-api-access-nrbw7\") pod \"maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg\" (UID: \"b8353165-d9b4-445a-9a11-5fdf04307860\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" Oct 31 03:36:05.483240 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.483218 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8353165-d9b4-445a-9a11-5fdf04307860-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg\" (UID: \"b8353165-d9b4-445a-9a11-5fdf04307860\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" Oct 31 03:36:05.483617 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.483589 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8353165-d9b4-445a-9a11-5fdf04307860-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg\" (UID: \"b8353165-d9b4-445a-9a11-5fdf04307860\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" Oct 31 03:36:05.600960 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.587169 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8353165-d9b4-445a-9a11-5fdf04307860-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg\" (UID: \"b8353165-d9b4-445a-9a11-5fdf04307860\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" Oct 31 03:36:05.600960 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.600590 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8353165-d9b4-445a-9a11-5fdf04307860-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg\" (UID: \"b8353165-d9b4-445a-9a11-5fdf04307860\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" Oct 31 03:36:05.600960 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.600629 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrbw7\" (UniqueName: \"kubernetes.io/projected/b8353165-d9b4-445a-9a11-5fdf04307860-kube-api-access-nrbw7\") pod \"maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg\" (UID: \"b8353165-d9b4-445a-9a11-5fdf04307860\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" Oct 31 03:36:05.600960 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.600739 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8353165-d9b4-445a-9a11-5fdf04307860-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg\" (UID: \"b8353165-d9b4-445a-9a11-5fdf04307860\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" Oct 31 03:36:05.602069 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.599639 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8353165-d9b4-445a-9a11-5fdf04307860-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg\" (UID: \"b8353165-d9b4-445a-9a11-5fdf04307860\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" Oct 31 03:36:05.602324 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.602287 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8353165-d9b4-445a-9a11-5fdf04307860-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg\" (UID: \"b8353165-d9b4-445a-9a11-5fdf04307860\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" Oct 31 03:36:05.614703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.605477 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8353165-d9b4-445a-9a11-5fdf04307860-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg\" (UID: \"b8353165-d9b4-445a-9a11-5fdf04307860\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" Oct 31 03:36:05.668708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.644340 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrbw7\" (UniqueName: \"kubernetes.io/projected/b8353165-d9b4-445a-9a11-5fdf04307860-kube-api-access-nrbw7\") pod \"maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg\" (UID: \"b8353165-d9b4-445a-9a11-5fdf04307860\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" Oct 31 03:36:05.691221 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.690187 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:05.822328 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:05.822293 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" Oct 31 03:36:06.722857 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:06.721807 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:36:06.766792 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:06.756630 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:36:07.217708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:07.217607 5490 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:36:07.218605 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:07.217961 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" podUID="50589d85-59be-4355-98ce-d7fd4cd6d331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:36:07.519760 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:07.516545 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:07.837749 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:07.820415 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:07.901512 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:07.841719 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg"] Oct 31 03:36:07.902085 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:07.901857 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:36:07.904216 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:07.904197 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" Oct 31 03:36:07.906006 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:07.905987 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:36:07.907890 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:07.907866 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:36:08.570125 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:08.544080 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:08.903865 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:08.903822 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" event={"ID":"b8353165-d9b4-445a-9a11-5fdf04307860","Type":"ContainerStarted","Data":"588ea823d809e73e387ebb9713e32ba6c9e4aa18c019b7a332dfc166259a4490"} Oct 31 03:36:08.904487 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:08.904458 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" event={"ID":"b8353165-d9b4-445a-9a11-5fdf04307860","Type":"ContainerStarted","Data":"e83f449fb0aa807abdfafa20285861ffc5da367660c4bcbbe1ae63e468f7744f"} Oct 31 03:36:08.904879 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:08.904642 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" Oct 31 03:36:09.450310 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:09.450274 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:09.881124 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:09.881016 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:10.410110 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:10.410030 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:11.287717 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:11.277322 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:12.101270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:12.101229 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:12.764606 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:12.764555 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:13.327591 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:13.327541 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:13.463260 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:13.463210 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:13.672737 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:13.672697 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:14.370063 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:14.369971 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:14.912725 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:14.912694 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" Oct 31 03:36:14.933000 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:14.932925 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" podStartSLOduration=10.308423505 podStartE2EDuration="10.932904719s" totalImagesPullingTime="624.481214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:08.924238105 +0000 UTC m=+1203.458933061" watchObservedRunningTime="2026-10-31 03:36:14.932904719 +0000 UTC m=+1209.467599674" Oct 31 03:36:15.120736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:15.120596 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:16.107024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:16.106980 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:16.394952 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:16.394869 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:16.531228 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:16.531194 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:16.676256 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:16.676212 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:16.825538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:16.824611 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:17.689012 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:17.688967 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:18.145397 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:36:18.145300 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:36:18.384321 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:18.383879 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:18.988751 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:18.988718 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:19.678742 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:19.676739 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:20.492465 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:20.492373 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:21.326442 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:21.326399 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:22.029922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:22.029837 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:22.799701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:22.799639 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:23.453692 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:23.453630 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:23.600441 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:23.600389 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:23.626769 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:23.625772 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:36:23.626769 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:23.625840 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" podUID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:36:23.749754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:23.749178 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:23.980402 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:23.980370 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:24.110015 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:24.109069 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:24.306311 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:24.306244 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:24.789373 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:24.787917 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:25.271538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:25.271494 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:25.532803 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:25.532412 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:25.871371 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:25.871261 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:26.115250 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:26.115205 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:26.388644 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:26.388607 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:26.467847 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:26.467798 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:36:26.468064 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:26.467880 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" podUID="a8a2a69d-e4b7-4338-a7b4-22042b94432f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:36:26.534593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:26.534556 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:27.102284 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:27.102175 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:27.886223 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:27.886187 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:28.193219 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:28.193183 5490 generic.go:356] "Generic (PLEG): container finished" podID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" containerID="13b21d79d292a339aad82c96083618172b35ee0058c235ccc96c8e15faa8a955" exitCode=0 Oct 31 03:36:28.193752 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:28.193262 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" event={"ID":"f91c38ef-6298-4315-9a8f-bedd67fcdd1c","Type":"ContainerDied","Data":"13b21d79d292a339aad82c96083618172b35ee0058c235ccc96c8e15faa8a955"} Oct 31 03:36:28.193890 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:28.193856 5490 scope.go:122] "RemoveContainer" containerID="13b21d79d292a339aad82c96083618172b35ee0058c235ccc96c8e15faa8a955" Oct 31 03:36:28.196825 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:28.196590 5490 generic.go:356] "Generic (PLEG): container finished" podID="a8a2a69d-e4b7-4338-a7b4-22042b94432f" containerID="b1fd0c247d873205c8e76e9c6e1b1c75b4d4095c36cc0b09c203f58771e7431c" exitCode=0 Oct 31 03:36:28.196953 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:28.196937 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" event={"ID":"a8a2a69d-e4b7-4338-a7b4-22042b94432f","Type":"ContainerDied","Data":"b1fd0c247d873205c8e76e9c6e1b1c75b4d4095c36cc0b09c203f58771e7431c"} Oct 31 03:36:28.197014 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:28.196960 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:36:28.199561 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:28.197125 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:36:28.199561 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:28.197230 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:36:28.199561 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:28.197846 5490 scope.go:122] "RemoveContainer" containerID="b1fd0c247d873205c8e76e9c6e1b1c75b4d4095c36cc0b09c203f58771e7431c" Oct 31 03:36:28.201201 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:28.201163 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:36:28.201333 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:28.201314 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:36:28.201507 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:28.201416 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:36:28.690719 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:28.690386 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:29.211341 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:29.210301 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" event={"ID":"f91c38ef-6298-4315-9a8f-bedd67fcdd1c","Type":"ContainerStarted","Data":"17c7ea0c1b9baf2b2884295852991be2d2d02e408dbf636ae882a55320ae0c09"} Oct 31 03:36:29.213755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:29.213210 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" event={"ID":"a8a2a69d-e4b7-4338-a7b4-22042b94432f","Type":"ContainerStarted","Data":"4d01c9e993f57b94598c5d6e7b7c4abb8bc04cae0370ec6131a24bcfede02400"} Oct 31 03:36:29.338714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:29.336186 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:29.399585 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:29.399544 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:36:29.399844 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:29.399802 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 03:36:29.399932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:29.399862 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" podUID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 03:36:29.648755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:29.606797 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:36:29.648755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:29.607216 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 03:36:29.648755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:29.607282 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" podUID="a8a2a69d-e4b7-4338-a7b4-22042b94432f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 03:36:29.648755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:29.608464 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:29.789733 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:29.788482 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:29.951100 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:29.947978 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:30.399391 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:30.399288 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 03:36:30.399391 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:30.399354 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" podUID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 03:36:30.585550 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:30.585517 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 03:36:30.585766 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:30.585571 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" podUID="a8a2a69d-e4b7-4338-a7b4-22042b94432f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 03:36:30.617306 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:30.617265 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:31.112159 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:31.112119 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:31.409395 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:31.409357 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:31.423880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:31.410206 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 03:36:31.424167 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:31.424137 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" podUID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 03:36:31.697919 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:31.697805 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 03:36:31.697919 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:31.697883 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" podUID="a8a2a69d-e4b7-4338-a7b4-22042b94432f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 03:36:31.889643 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:31.889603 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:32.400377 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:32.400235 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 03:36:32.400377 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:32.400292 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" podUID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 03:36:32.585714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:32.585648 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 03:36:32.586124 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:32.585747 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" podUID="a8a2a69d-e4b7-4338-a7b4-22042b94432f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 03:36:32.862704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:32.862025 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:33.171702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:33.169715 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:33.399456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:33.399420 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 03:36:33.399715 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:33.399487 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" podUID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 03:36:33.584986 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:33.584888 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 03:36:33.585252 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:33.585225 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" podUID="a8a2a69d-e4b7-4338-a7b4-22042b94432f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 03:36:33.812781 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:33.812748 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:34.399861 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:34.399826 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 31 03:36:34.400072 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:34.399901 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" podUID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 31 03:36:34.585604 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:34.585568 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 03:36:34.585822 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:34.585620 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" podUID="a8a2a69d-e4b7-4338-a7b4-22042b94432f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 03:36:34.616625 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:34.616466 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:35.421699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:35.417046 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:35.429464 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:35.429422 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:36:35.430077 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:35.430051 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:36:35.431190 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:35.431166 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:36:35.589996 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:35.589953 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:36:35.591869 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:35.591812 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:36:35.592562 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:35.592537 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 03:36:36.081698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:36.073293 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:36.755508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:36.755469 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:36.905398 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:36.905358 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:37.495400 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:37.495351 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:38.241607 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:38.241549 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:38.784399 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:38.784340 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:39.416146 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:39.416089 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:40.091803 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:40.091763 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:40.810864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:40.810815 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:40.943970 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:40.943927 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:41.471142 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:41.471092 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:42.166238 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:42.164403 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:42.883497 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:42.883457 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:43.715707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:43.713350 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:44.687476 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:44.686689 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:45.597065 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:45.597013 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:46.841244 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:46.840526 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:47.320661 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:36:47.320612 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:36:47.740703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:47.721420 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:36:47.790707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:47.738841 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:36:47.790707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:47.739215 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:36:48.206368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:48.206326 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:48.244754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:48.243805 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:36:48.273902 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:48.273847 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-k9bcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:36:48.274785 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:48.274244 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" podUID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:36:48.301703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:48.297741 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:36:48.467099 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:48.466998 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:36:49.327000 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:49.326934 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:50.075201 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:50.075156 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:50.751950 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:50.751828 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:51.344322 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:51.344281 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:51.914444 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:51.914392 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:52.594422 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:52.594384 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:53.192081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:53.192039 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:53.806585 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:53.806546 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:54.661352 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:54.661307 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:55.045794 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:55.045627 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:55.589829 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:55.589792 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:55.725902 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:55.725864 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:56.207284 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:56.207245 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:56.425747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:56.408019 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:56.857019 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:56.856979 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:57.525515 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:57.525458 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:58.214268 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:58.214227 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:58.921475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:58.921439 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:36:59.877342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:36:59.877299 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:00.111847 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:00.111804 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:00.245462 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:00.245431 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:00.590556 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:00.590448 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:01.374169 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:01.374122 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:01.601812 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:01.601773 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:01.779697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:01.776277 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:02.391237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:02.391198 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:03.097867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:03.097772 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:03.695591 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:03.695545 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:04.305354 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:04.305315 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:05.096992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:05.096953 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:05.550021 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:05.549983 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:05.666549 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:05.666510 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b"] Oct 31 03:37:05.672879 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:05.671290 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" podUID="6747ac36-d4a2-4a74-a97c-2c53a909da9d" containerName="istio-proxy" containerID="cri-o://384285a7d4ba0dd08417683c21bbdf4b4d2ff420e9c2f12ae2698493c96d8196" gracePeriod=30 Oct 31 03:37:05.903815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:05.903778 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:06.125705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:06.121994 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:06.352037 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:06.344281 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:06.901777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:06.901739 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:07.796663 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:07.796632 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:08.018410 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:08.018361 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:08.659631 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:08.659529 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:09.484430 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:09.484300 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:10.097160 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:10.097105 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:10.281772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:10.281732 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:10.687034 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:10.683329 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:11.279115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.279073 5490 generic.go:356] "Generic (PLEG): container finished" podID="6747ac36-d4a2-4a74-a97c-2c53a909da9d" containerID="384285a7d4ba0dd08417683c21bbdf4b4d2ff420e9c2f12ae2698493c96d8196" exitCode=0 Oct 31 03:37:11.279444 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.279361 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" event={"ID":"6747ac36-d4a2-4a74-a97c-2c53a909da9d","Type":"ContainerDied","Data":"384285a7d4ba0dd08417683c21bbdf4b4d2ff420e9c2f12ae2698493c96d8196"} Oct 31 03:37:11.314444 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.314385 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:11.453940 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.453898 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:11.620798 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.620560 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:11.980336 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.980301 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:37:11.981063 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.981038 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istiod-ca-cert\") pod \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " Oct 31 03:37:11.981326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.981291 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-workload-socket\") pod \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " Oct 31 03:37:11.981524 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.981364 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-data\") pod \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " Oct 31 03:37:11.981524 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.981428 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6747ac36-d4a2-4a74-a97c-2c53a909da9d-kube-api-access-ttthq\" (UniqueName: \"kubernetes.io/projected/6747ac36-d4a2-4a74-a97c-2c53a909da9d-kube-api-access-ttthq\") pod \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " Oct 31 03:37:11.981661 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.981536 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-podinfo\") pod \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " Oct 31 03:37:11.981661 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.981601 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-workload-certs\") pod \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " Oct 31 03:37:11.981798 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.981666 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-token\" (UniqueName: \"kubernetes.io/projected/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-token\") pod \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " Oct 31 03:37:11.981798 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.981767 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-envoy\") pod \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " Oct 31 03:37:11.981798 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.981789 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-credential-socket\") pod \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\" (UID: \"6747ac36-d4a2-4a74-a97c-2c53a909da9d\") " Oct 31 03:37:11.985584 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.985317 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-credential-socket" pod "6747ac36-d4a2-4a74-a97c-2c53a909da9d" (UID: "6747ac36-d4a2-4a74-a97c-2c53a909da9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:11.985584 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.985543 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-workload-certs" pod "6747ac36-d4a2-4a74-a97c-2c53a909da9d" (UID: "6747ac36-d4a2-4a74-a97c-2c53a909da9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:11.985584 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.985571 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istiod-ca-cert" pod "6747ac36-d4a2-4a74-a97c-2c53a909da9d" (UID: "6747ac36-d4a2-4a74-a97c-2c53a909da9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:37:11.985901 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.985868 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-data" pod "6747ac36-d4a2-4a74-a97c-2c53a909da9d" (UID: "6747ac36-d4a2-4a74-a97c-2c53a909da9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:11.986256 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.986235 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-workload-socket" pod "6747ac36-d4a2-4a74-a97c-2c53a909da9d" (UID: "6747ac36-d4a2-4a74-a97c-2c53a909da9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:11.989141 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.989108 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-podinfo" pod "6747ac36-d4a2-4a74-a97c-2c53a909da9d" (UID: "6747ac36-d4a2-4a74-a97c-2c53a909da9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:37:11.989235 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.989165 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-token" pod "6747ac36-d4a2-4a74-a97c-2c53a909da9d" (UID: "6747ac36-d4a2-4a74-a97c-2c53a909da9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:11.989235 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.989171 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-envoy" pod "6747ac36-d4a2-4a74-a97c-2c53a909da9d" (UID: "6747ac36-d4a2-4a74-a97c-2c53a909da9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:11.989235 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:11.989220 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6747ac36-d4a2-4a74-a97c-2c53a909da9d-kube-api-access-ttthq" pod "6747ac36-d4a2-4a74-a97c-2c53a909da9d" (UID: "6747ac36-d4a2-4a74-a97c-2c53a909da9d"). InnerVolumeSpecName "kube-api-access-ttthq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:12.083850 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.083811 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:12.083850 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.083849 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:12.084127 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.083865 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:12.084127 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.083880 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:12.084127 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.083895 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:12.084127 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.083909 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:12.084127 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.083922 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttthq\" (UniqueName: \"kubernetes.io/projected/6747ac36-d4a2-4a74-a97c-2c53a909da9d-kube-api-access-ttthq\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:12.084127 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.083937 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6747ac36-d4a2-4a74-a97c-2c53a909da9d-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:12.084127 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.083952 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6747ac36-d4a2-4a74-a97c-2c53a909da9d-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:12.292867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.292778 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" event={"ID":"6747ac36-d4a2-4a74-a97c-2c53a909da9d","Type":"ContainerDied","Data":"c873da69502e6c79b1cfe9dfdb6ca621ee2a5b7386ec1c63ee1e681e71934d34"} Oct 31 03:37:12.292867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.292864 5490 scope.go:122] "RemoveContainer" containerID="384285a7d4ba0dd08417683c21bbdf4b4d2ff420e9c2f12ae2698493c96d8196" Oct 31 03:37:12.293041 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.293032 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b" Oct 31 03:37:12.294970 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.294939 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r"] Oct 31 03:37:12.296174 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.296154 5490 cpu_manager.go:414] "containerMap: removing container" podUID="6747ac36-d4a2-4a74-a97c-2c53a909da9d" containerName="istio-proxy" Oct 31 03:37:12.296265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.296181 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6747ac36-d4a2-4a74-a97c-2c53a909da9d" containerName="istio-proxy" Oct 31 03:37:12.296450 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.296437 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="6747ac36-d4a2-4a74-a97c-2c53a909da9d" containerName="istio-proxy" Oct 31 03:37:12.301412 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.301390 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" Oct 31 03:37:12.310292 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.309374 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b87769-serving-cert\"" Oct 31 03:37:12.318127 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.316981 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r"] Oct 31 03:37:12.350040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.349981 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b"] Oct 31 03:37:12.354373 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.354339 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-zxf4b"] Oct 31 03:37:12.385593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.385562 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:12.389143 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.389115 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efde673-8a62-4eb0-b2cd-3fad992e1867-openshift-service-ca\") pod \"maas-api-e2e-api-b-b87769-649bdd7ff5-q758r\" (UID: \"1efde673-8a62-4eb0-b2cd-3fad992e1867\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" Oct 31 03:37:12.389411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.389392 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb84b\" (UniqueName: \"kubernetes.io/projected/1efde673-8a62-4eb0-b2cd-3fad992e1867-kube-api-access-xb84b\") pod \"maas-api-e2e-api-b-b87769-649bdd7ff5-q758r\" (UID: \"1efde673-8a62-4eb0-b2cd-3fad992e1867\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" Oct 31 03:37:12.389494 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.389449 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1efde673-8a62-4eb0-b2cd-3fad992e1867-maas-api-tls\") pod \"maas-api-e2e-api-b-b87769-649bdd7ff5-q758r\" (UID: \"1efde673-8a62-4eb0-b2cd-3fad992e1867\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" Oct 31 03:37:12.389555 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.389541 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1efde673-8a62-4eb0-b2cd-3fad992e1867-metrics-certs\") pod \"maas-api-e2e-api-b-b87769-649bdd7ff5-q758r\" (UID: \"1efde673-8a62-4eb0-b2cd-3fad992e1867\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" Oct 31 03:37:12.491579 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.491518 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb84b\" (UniqueName: \"kubernetes.io/projected/1efde673-8a62-4eb0-b2cd-3fad992e1867-kube-api-access-xb84b\") pod \"maas-api-e2e-api-b-b87769-649bdd7ff5-q758r\" (UID: \"1efde673-8a62-4eb0-b2cd-3fad992e1867\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" Oct 31 03:37:12.491822 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.491633 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1efde673-8a62-4eb0-b2cd-3fad992e1867-maas-api-tls\") pod \"maas-api-e2e-api-b-b87769-649bdd7ff5-q758r\" (UID: \"1efde673-8a62-4eb0-b2cd-3fad992e1867\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" Oct 31 03:37:12.491822 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.491748 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1efde673-8a62-4eb0-b2cd-3fad992e1867-metrics-certs\") pod \"maas-api-e2e-api-b-b87769-649bdd7ff5-q758r\" (UID: \"1efde673-8a62-4eb0-b2cd-3fad992e1867\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" Oct 31 03:37:12.491822 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:37:12.491762 5490 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-serving-cert: secret "maas-api-e2e-api-b-b87769-serving-cert" not found Oct 31 03:37:12.492027 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:37:12.491849 5490 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1efde673-8a62-4eb0-b2cd-3fad992e1867-maas-api-tls podName:1efde673-8a62-4eb0-b2cd-3fad992e1867 nodeName:}" failed. No retries permitted until 2026-10-31 03:37:12.991816083 +0000 UTC m=+1267.526511035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1efde673-8a62-4eb0-b2cd-3fad992e1867-maas-api-tls") pod "maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" (UID: "1efde673-8a62-4eb0-b2cd-3fad992e1867") : secret "maas-api-e2e-api-b-b87769-serving-cert" not found Oct 31 03:37:12.492027 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.491999 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efde673-8a62-4eb0-b2cd-3fad992e1867-openshift-service-ca\") pod \"maas-api-e2e-api-b-b87769-649bdd7ff5-q758r\" (UID: \"1efde673-8a62-4eb0-b2cd-3fad992e1867\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" Oct 31 03:37:12.492669 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.492647 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efde673-8a62-4eb0-b2cd-3fad992e1867-openshift-service-ca\") pod \"maas-api-e2e-api-b-b87769-649bdd7ff5-q758r\" (UID: \"1efde673-8a62-4eb0-b2cd-3fad992e1867\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" Oct 31 03:37:12.494963 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.494942 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1efde673-8a62-4eb0-b2cd-3fad992e1867-metrics-certs\") pod \"maas-api-e2e-api-b-b87769-649bdd7ff5-q758r\" (UID: \"1efde673-8a62-4eb0-b2cd-3fad992e1867\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" Oct 31 03:37:12.501336 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.501262 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb84b\" (UniqueName: \"kubernetes.io/projected/1efde673-8a62-4eb0-b2cd-3fad992e1867-kube-api-access-xb84b\") pod \"maas-api-e2e-api-b-b87769-649bdd7ff5-q758r\" (UID: \"1efde673-8a62-4eb0-b2cd-3fad992e1867\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" Oct 31 03:37:12.524852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.524777 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:12.800336 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:12.800290 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:13.004869 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:13.004833 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1efde673-8a62-4eb0-b2cd-3fad992e1867-maas-api-tls\") pod \"maas-api-e2e-api-b-b87769-649bdd7ff5-q758r\" (UID: \"1efde673-8a62-4eb0-b2cd-3fad992e1867\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" Oct 31 03:37:13.007429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:13.007404 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1efde673-8a62-4eb0-b2cd-3fad992e1867-maas-api-tls\") pod \"maas-api-e2e-api-b-b87769-649bdd7ff5-q758r\" (UID: \"1efde673-8a62-4eb0-b2cd-3fad992e1867\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" Oct 31 03:37:13.235217 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:13.235183 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" Oct 31 03:37:13.352571 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:13.352471 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:13.536845 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:13.536814 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r"] Oct 31 03:37:13.537506 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:37:13.537475 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1efde673_8a62_4eb0_b2cd_3fad992e1867.slice/crio-dfd8b6ee4ada11f43d48fb8b939f2c716cb10961deeaa837a713cffa32833760 WatchSource:0}: Error finding container dfd8b6ee4ada11f43d48fb8b939f2c716cb10961deeaa837a713cffa32833760: Status 404 returned error can't find the container with id dfd8b6ee4ada11f43d48fb8b939f2c716cb10961deeaa837a713cffa32833760 Oct 31 03:37:13.539275 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:13.539253 5490 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:37:13.579632 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:13.579598 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6747ac36-d4a2-4a74-a97c-2c53a909da9d" path="/var/lib/kubelet/pods/6747ac36-d4a2-4a74-a97c-2c53a909da9d/volumes" Oct 31 03:37:14.113378 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:14.111130 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:14.316964 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:14.316862 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" event={"ID":"1efde673-8a62-4eb0-b2cd-3fad992e1867","Type":"ContainerStarted","Data":"b3e8db7d3f0c06d19c253e0b08a6131bb0777971c39f28e6fb329bdada2b0c25"} Oct 31 03:37:14.316964 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:14.316927 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" event={"ID":"1efde673-8a62-4eb0-b2cd-3fad992e1867","Type":"ContainerStarted","Data":"dfd8b6ee4ada11f43d48fb8b939f2c716cb10961deeaa837a713cffa32833760"} Oct 31 03:37:14.317206 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:14.316964 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" Oct 31 03:37:14.343631 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:14.343563 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" podStartSLOduration=1.943659316 podStartE2EDuration="2.343545367s" totalImagesPullingTime="399.886051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:14.335206804 +0000 UTC m=+1268.869901763" watchObservedRunningTime="2026-10-31 03:37:14.343545367 +0000 UTC m=+1268.878240322" Oct 31 03:37:14.851302 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:14.851261 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:15.031417 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:15.031368 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:15.080738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:15.066271 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx"] Oct 31 03:37:15.099024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:15.098004 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" podUID="50589d85-59be-4355-98ce-d7fd4cd6d331" containerName="istio-proxy" containerID="cri-o://d771073eec5f57ef43e166193854180fdc1704f1dd9b1675384528c754d51c4c" gracePeriod=30 Oct 31 03:37:15.170902 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:15.167914 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:15.315700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:15.315158 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:16.298940 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:16.298831 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:16.603880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:16.603792 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:17.151769 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:37:17.150260 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:37:17.380209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:17.380143 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:18.005359 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:18.005321 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:18.701773 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:18.695498 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:19.314031 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:19.313995 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:19.800832 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:19.800795 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:19.934761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:19.934728 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:20.066772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.064331 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:20.432289 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.432265 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" Oct 31 03:37:20.488567 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.488530 5490 generic.go:356] "Generic (PLEG): container finished" podID="50589d85-59be-4355-98ce-d7fd4cd6d331" containerID="d771073eec5f57ef43e166193854180fdc1704f1dd9b1675384528c754d51c4c" exitCode=0 Oct 31 03:37:20.488754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.488561 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" event={"ID":"50589d85-59be-4355-98ce-d7fd4cd6d331","Type":"ContainerDied","Data":"d771073eec5f57ef43e166193854180fdc1704f1dd9b1675384528c754d51c4c"} Oct 31 03:37:20.559311 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.559285 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:37:20.648982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.648892 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-workload-certs\") pod \"50589d85-59be-4355-98ce-d7fd4cd6d331\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " Oct 31 03:37:20.648982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.648976 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-workload-socket\") pod \"50589d85-59be-4355-98ce-d7fd4cd6d331\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " Oct 31 03:37:20.649209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.649007 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-podinfo\") pod \"50589d85-59be-4355-98ce-d7fd4cd6d331\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " Oct 31 03:37:20.649209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.649041 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-credential-socket\") pod \"50589d85-59be-4355-98ce-d7fd4cd6d331\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " Oct 31 03:37:20.649209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.649068 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50589d85-59be-4355-98ce-d7fd4cd6d331-kube-api-access-tmt29\" (UniqueName: \"kubernetes.io/projected/50589d85-59be-4355-98ce-d7fd4cd6d331-kube-api-access-tmt29\") pod \"50589d85-59be-4355-98ce-d7fd4cd6d331\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " Oct 31 03:37:20.649209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.649106 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-token\" (UniqueName: \"kubernetes.io/projected/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-token\") pod \"50589d85-59be-4355-98ce-d7fd4cd6d331\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " Oct 31 03:37:20.649345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.649218 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-envoy\") pod \"50589d85-59be-4355-98ce-d7fd4cd6d331\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " Oct 31 03:37:20.649345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.649283 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-data\") pod \"50589d85-59be-4355-98ce-d7fd4cd6d331\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " Oct 31 03:37:20.649412 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.649379 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50589d85-59be-4355-98ce-d7fd4cd6d331-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50589d85-59be-4355-98ce-d7fd4cd6d331-istiod-ca-cert\") pod \"50589d85-59be-4355-98ce-d7fd4cd6d331\" (UID: \"50589d85-59be-4355-98ce-d7fd4cd6d331\") " Oct 31 03:37:20.650990 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.650950 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50589d85-59be-4355-98ce-d7fd4cd6d331-istiod-ca-cert" pod "50589d85-59be-4355-98ce-d7fd4cd6d331" (UID: "50589d85-59be-4355-98ce-d7fd4cd6d331"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:37:20.651282 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.651256 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-workload-certs" pod "50589d85-59be-4355-98ce-d7fd4cd6d331" (UID: "50589d85-59be-4355-98ce-d7fd4cd6d331"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:20.651718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.651655 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-workload-socket" pod "50589d85-59be-4355-98ce-d7fd4cd6d331" (UID: "50589d85-59be-4355-98ce-d7fd4cd6d331"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:20.653046 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.653017 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-data" pod "50589d85-59be-4355-98ce-d7fd4cd6d331" (UID: "50589d85-59be-4355-98ce-d7fd4cd6d331"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:20.653441 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.653285 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-credential-socket" pod "50589d85-59be-4355-98ce-d7fd4cd6d331" (UID: "50589d85-59be-4355-98ce-d7fd4cd6d331"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:20.654942 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.654910 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-envoy" pod "50589d85-59be-4355-98ce-d7fd4cd6d331" (UID: "50589d85-59be-4355-98ce-d7fd4cd6d331"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:20.656369 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.656328 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50589d85-59be-4355-98ce-d7fd4cd6d331-kube-api-access-tmt29" pod "50589d85-59be-4355-98ce-d7fd4cd6d331" (UID: "50589d85-59be-4355-98ce-d7fd4cd6d331"). InnerVolumeSpecName "kube-api-access-tmt29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:20.657229 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.657194 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-token" pod "50589d85-59be-4355-98ce-d7fd4cd6d331" (UID: "50589d85-59be-4355-98ce-d7fd4cd6d331"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:20.668941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.668902 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-podinfo" pod "50589d85-59be-4355-98ce-d7fd4cd6d331" (UID: "50589d85-59be-4355-98ce-d7fd4cd6d331"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:37:20.753293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.753253 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:20.753293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.753292 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:20.753555 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.753307 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:20.753555 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.753323 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:20.753555 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.753338 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmt29\" (UniqueName: \"kubernetes.io/projected/50589d85-59be-4355-98ce-d7fd4cd6d331-kube-api-access-tmt29\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:20.753555 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.753352 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:20.753555 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.753369 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:20.753555 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.753387 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50589d85-59be-4355-98ce-d7fd4cd6d331-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:20.753555 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.753403 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50589d85-59be-4355-98ce-d7fd4cd6d331-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:20.816545 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.816506 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2"] Oct 31 03:37:20.817041 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.816714 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" podUID="0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" containerName="istio-proxy" containerID="cri-o://8c00d36c8cc8ea0bb4deb8d74016fa1f87e9b2d0b188a91405ec1d54aa875bcd" gracePeriod=30 Oct 31 03:37:20.882386 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.882348 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv"] Oct 31 03:37:20.889660 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.889623 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" podUID="5aa4a40b-944a-4113-9e1e-50c844f77cb4" containerName="istio-proxy" containerID="cri-o://ae26df749e7769606d444bf73ea3267a4b28f81ea175662e7c431999ab42e759" gracePeriod=30 Oct 31 03:37:20.908773 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:20.908740 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:21.499640 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:21.499604 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" Oct 31 03:37:21.499640 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:21.499633 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx" event={"ID":"50589d85-59be-4355-98ce-d7fd4cd6d331","Type":"ContainerDied","Data":"ce81f2e75a7ad45ffb1ad69e4ca14b931baa5f012f9c4686721db46d9aa7fd3f"} Oct 31 03:37:21.499905 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:21.499719 5490 scope.go:122] "RemoveContainer" containerID="d771073eec5f57ef43e166193854180fdc1704f1dd9b1675384528c754d51c4c" Oct 31 03:37:21.504666 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:21.504633 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:21.537484 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:21.537446 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx"] Oct 31 03:37:21.544634 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:21.544581 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-bj4nx"] Oct 31 03:37:21.579139 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:21.579107 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50589d85-59be-4355-98ce-d7fd4cd6d331" path="/var/lib/kubelet/pods/50589d85-59be-4355-98ce-d7fd4cd6d331/volumes" Oct 31 03:37:22.127106 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:22.125790 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:22.800509 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:22.800245 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:23.414204 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.414169 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz"] Oct 31 03:37:23.415067 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.415048 5490 cpu_manager.go:414] "containerMap: removing container" podUID="50589d85-59be-4355-98ce-d7fd4cd6d331" containerName="istio-proxy" Oct 31 03:37:23.415132 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.415068 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50589d85-59be-4355-98ce-d7fd4cd6d331" containerName="istio-proxy" Oct 31 03:37:23.415238 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.415226 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="50589d85-59be-4355-98ce-d7fd4cd6d331" containerName="istio-proxy" Oct 31 03:37:23.419079 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.419060 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" Oct 31 03:37:23.429751 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.429725 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-afb8be-serving-cert\"" Oct 31 03:37:23.438601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.438561 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz"] Oct 31 03:37:23.511620 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.511572 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:23.513923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.513895 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz\" (UID: \"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" Oct 31 03:37:23.514060 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.514041 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz\" (UID: \"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" Oct 31 03:37:23.514121 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.514103 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz\" (UID: \"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" Oct 31 03:37:23.514264 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.514241 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7qrt\" (UniqueName: \"kubernetes.io/projected/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-kube-api-access-z7qrt\") pod \"maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz\" (UID: \"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" Oct 31 03:37:23.616122 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.616081 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz\" (UID: \"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" Oct 31 03:37:23.616333 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.616161 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz\" (UID: \"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" Oct 31 03:37:23.616333 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.616293 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7qrt\" (UniqueName: \"kubernetes.io/projected/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-kube-api-access-z7qrt\") pod \"maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz\" (UID: \"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" Oct 31 03:37:23.616599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.616540 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz\" (UID: \"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" Oct 31 03:37:23.617470 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.617443 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz\" (UID: \"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" Oct 31 03:37:23.625247 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.625214 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz\" (UID: \"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" Oct 31 03:37:23.645031 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.644996 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7qrt\" (UniqueName: \"kubernetes.io/projected/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-kube-api-access-z7qrt\") pod \"maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz\" (UID: \"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" Oct 31 03:37:23.652202 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.652135 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz\" (UID: \"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" Oct 31 03:37:23.662354 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.662318 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:23.744159 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:23.744069 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" Oct 31 03:37:24.166639 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:24.166609 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz"] Oct 31 03:37:24.528972 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:24.528933 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" event={"ID":"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0","Type":"ContainerStarted","Data":"5b6efb82677df3141f5bf8e4caab2b1402c52d4d39217a4e5bd118be49280cf3"} Oct 31 03:37:24.564226 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:24.564168 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:25.165941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:25.165899 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:25.539011 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:25.538907 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" event={"ID":"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0","Type":"ContainerStarted","Data":"ddbdebadd3ab577b885104a38dcbb9fd9a3a9c5ae4d69ef6f0f19ef3883821a7"} Oct 31 03:37:25.539011 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:25.538989 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" Oct 31 03:37:25.557093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:25.557044 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" podStartSLOduration=2.117937929 podStartE2EDuration="2.557028657s" totalImagesPullingTime="439.090728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:25.555173759 +0000 UTC m=+1280.089868714" watchObservedRunningTime="2026-10-31 03:37:25.557028657 +0000 UTC m=+1280.091723650" Oct 31 03:37:26.017731 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:25.984311 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:26.443099 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.443066 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:37:26.549801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.549768 5490 generic.go:356] "Generic (PLEG): container finished" podID="5aa4a40b-944a-4113-9e1e-50c844f77cb4" containerID="ae26df749e7769606d444bf73ea3267a4b28f81ea175662e7c431999ab42e759" exitCode=0 Oct 31 03:37:26.550264 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.549853 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" event={"ID":"5aa4a40b-944a-4113-9e1e-50c844f77cb4","Type":"ContainerDied","Data":"ae26df749e7769606d444bf73ea3267a4b28f81ea175662e7c431999ab42e759"} Oct 31 03:37:26.551920 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.551883 5490 generic.go:356] "Generic (PLEG): container finished" podID="0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" containerID="8c00d36c8cc8ea0bb4deb8d74016fa1f87e9b2d0b188a91405ec1d54aa875bcd" exitCode=0 Oct 31 03:37:26.552057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.551964 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" Oct 31 03:37:26.552057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.551973 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" event={"ID":"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9","Type":"ContainerDied","Data":"8c00d36c8cc8ea0bb4deb8d74016fa1f87e9b2d0b188a91405ec1d54aa875bcd"} Oct 31 03:37:26.552057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.552038 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2" event={"ID":"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9","Type":"ContainerDied","Data":"16090692aeefcad852aa88f15bafa2d94b5ab7f39a2ee0a71112818e1a24d9b0"} Oct 31 03:37:26.552204 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.552070 5490 scope.go:122] "RemoveContainer" containerID="8c00d36c8cc8ea0bb4deb8d74016fa1f87e9b2d0b188a91405ec1d54aa875bcd" Oct 31 03:37:26.562252 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.562218 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-workload-certs\") pod \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " Oct 31 03:37:26.562541 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.562518 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-workload-certs" pod "0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" (UID: "0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:26.562875 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.562828 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-kube-api-access-qs4zx\" (UniqueName: \"kubernetes.io/projected/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-kube-api-access-qs4zx\") pod \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " Oct 31 03:37:26.563334 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.562875 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-podinfo\") pod \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " Oct 31 03:37:26.563334 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.562934 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-workload-socket\") pod \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " Oct 31 03:37:26.563334 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.563012 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istiod-ca-cert\") pod \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " Oct 31 03:37:26.563334 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.563125 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-credential-socket\") pod \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " Oct 31 03:37:26.563334 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.563157 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-envoy\") pod \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " Oct 31 03:37:26.563334 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.563229 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-token\" (UniqueName: \"kubernetes.io/projected/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-token\") pod \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " Oct 31 03:37:26.564077 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.563393 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-data\") pod \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\" (UID: \"0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9\") " Oct 31 03:37:26.565559 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.565335 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.565752 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.565715 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-data" pod "0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" (UID: "0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:26.566290 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.566262 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-credential-socket" pod "0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" (UID: "0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:26.566390 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.566327 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-workload-socket" pod "0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" (UID: "0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:26.566517 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.566494 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istiod-ca-cert" pod "0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" (UID: "0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:37:26.569106 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.569076 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-podinfo" pod "0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" (UID: "0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:37:26.572896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.571579 5490 scope.go:122] "RemoveContainer" containerID="8c00d36c8cc8ea0bb4deb8d74016fa1f87e9b2d0b188a91405ec1d54aa875bcd" Oct 31 03:37:26.574192 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:37:26.574090 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c00d36c8cc8ea0bb4deb8d74016fa1f87e9b2d0b188a91405ec1d54aa875bcd\": container with ID starting with 8c00d36c8cc8ea0bb4deb8d74016fa1f87e9b2d0b188a91405ec1d54aa875bcd not found: ID does not exist" containerID="8c00d36c8cc8ea0bb4deb8d74016fa1f87e9b2d0b188a91405ec1d54aa875bcd" Oct 31 03:37:26.574192 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.574129 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c00d36c8cc8ea0bb4deb8d74016fa1f87e9b2d0b188a91405ec1d54aa875bcd"} err="failed to get container status \"8c00d36c8cc8ea0bb4deb8d74016fa1f87e9b2d0b188a91405ec1d54aa875bcd\": rpc error: code = NotFound desc = could not find container \"8c00d36c8cc8ea0bb4deb8d74016fa1f87e9b2d0b188a91405ec1d54aa875bcd\": container with ID starting with 8c00d36c8cc8ea0bb4deb8d74016fa1f87e9b2d0b188a91405ec1d54aa875bcd not found: ID does not exist" Oct 31 03:37:26.577361 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.577319 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-kube-api-access-qs4zx" pod "0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" (UID: "0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9"). InnerVolumeSpecName "kube-api-access-qs4zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:26.590505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.590469 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-token" pod "0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" (UID: "0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:26.590757 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.590728 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-envoy" pod "0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" (UID: "0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:26.668059 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.668009 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.668059 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.668041 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.668059 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.668057 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.668441 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.668070 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.668441 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.668083 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.668441 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.668096 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs4zx\" (UniqueName: \"kubernetes.io/projected/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-kube-api-access-qs4zx\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.668441 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.668108 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.668441 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.668121 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.771312 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.771278 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:37:26.870421 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.870370 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa4a40b-944a-4113-9e1e-50c844f77cb4-kube-api-access-9v9h6\" (UniqueName: \"kubernetes.io/projected/5aa4a40b-944a-4113-9e1e-50c844f77cb4-kube-api-access-9v9h6\") pod \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " Oct 31 03:37:26.870589 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.870427 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istiod-ca-cert\") pod \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " Oct 31 03:37:26.870717 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.870696 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-podinfo\") pod \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " Oct 31 03:37:26.871552 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.870781 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-token\" (UniqueName: \"kubernetes.io/projected/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-token\") pod \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " Oct 31 03:37:26.871552 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.870809 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-workload-certs\") pod \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " Oct 31 03:37:26.871552 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.870914 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-workload-socket\") pod \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " Oct 31 03:37:26.871552 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.870994 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-envoy\") pod \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " Oct 31 03:37:26.871552 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.871091 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-data\") pod \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " Oct 31 03:37:26.871552 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.871204 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-credential-socket\") pod \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\" (UID: \"5aa4a40b-944a-4113-9e1e-50c844f77cb4\") " Oct 31 03:37:26.872257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.872213 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-credential-socket" pod "5aa4a40b-944a-4113-9e1e-50c844f77cb4" (UID: "5aa4a40b-944a-4113-9e1e-50c844f77cb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:26.872417 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.872392 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-workload-certs" pod "5aa4a40b-944a-4113-9e1e-50c844f77cb4" (UID: "5aa4a40b-944a-4113-9e1e-50c844f77cb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:26.872591 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.872426 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-data" pod "5aa4a40b-944a-4113-9e1e-50c844f77cb4" (UID: "5aa4a40b-944a-4113-9e1e-50c844f77cb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:26.872591 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.872477 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istiod-ca-cert" pod "5aa4a40b-944a-4113-9e1e-50c844f77cb4" (UID: "5aa4a40b-944a-4113-9e1e-50c844f77cb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:37:26.872852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.872594 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-workload-socket" pod "5aa4a40b-944a-4113-9e1e-50c844f77cb4" (UID: "5aa4a40b-944a-4113-9e1e-50c844f77cb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:26.875823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.875794 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa4a40b-944a-4113-9e1e-50c844f77cb4-kube-api-access-9v9h6" pod "5aa4a40b-944a-4113-9e1e-50c844f77cb4" (UID: "5aa4a40b-944a-4113-9e1e-50c844f77cb4"). InnerVolumeSpecName "kube-api-access-9v9h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:26.877155 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.877106 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-podinfo" pod "5aa4a40b-944a-4113-9e1e-50c844f77cb4" (UID: "5aa4a40b-944a-4113-9e1e-50c844f77cb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:37:26.877155 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.877113 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-envoy" pod "5aa4a40b-944a-4113-9e1e-50c844f77cb4" (UID: "5aa4a40b-944a-4113-9e1e-50c844f77cb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:37:26.878412 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.878389 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-token" pod "5aa4a40b-944a-4113-9e1e-50c844f77cb4" (UID: "5aa4a40b-944a-4113-9e1e-50c844f77cb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:37:26.916507 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.916474 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2"] Oct 31 03:37:26.920985 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.920957 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-whrx2"] Oct 31 03:37:26.953656 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.953562 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:26.978783 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.977663 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.978783 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.977718 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v9h6\" (UniqueName: \"kubernetes.io/projected/5aa4a40b-944a-4113-9e1e-50c844f77cb4-kube-api-access-9v9h6\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.978783 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.977738 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.978783 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.977753 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.978783 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.977767 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.978783 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.977780 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.978783 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.977794 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.978783 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.977808 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:26.978783 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:26.977823 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa4a40b-944a-4113-9e1e-50c844f77cb4-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:37:27.672375 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:27.672332 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" path="/var/lib/kubelet/pods/0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9/volumes" Oct 31 03:37:27.675880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:27.674407 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" Oct 31 03:37:27.677163 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:27.677132 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv" event={"ID":"5aa4a40b-944a-4113-9e1e-50c844f77cb4","Type":"ContainerDied","Data":"be642f4ac7ec42e76fb26c1600da989765514131162edae6fdd3353d1cbce473"} Oct 31 03:37:27.683814 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:27.683782 5490 scope.go:122] "RemoveContainer" containerID="ae26df749e7769606d444bf73ea3267a4b28f81ea175662e7c431999ab42e759" Oct 31 03:37:27.801819 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:27.801741 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv"] Oct 31 03:37:27.806946 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:27.806909 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-5bnfv"] Oct 31 03:37:27.935846 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:27.935764 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:28.377517 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:28.377429 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:28.707710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:28.706939 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:29.379648 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:29.379590 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:29.578548 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:29.578515 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aa4a40b-944a-4113-9e1e-50c844f77cb4" path="/var/lib/kubelet/pods/5aa4a40b-944a-4113-9e1e-50c844f77cb4/volumes" Oct 31 03:37:29.987639 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:29.987245 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:30.486918 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:30.486881 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:30.623456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:30.623377 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:31.345513 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:31.345475 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:31.492411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:31.492373 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:31.558169 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:31.558139 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" Oct 31 03:37:31.656758 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:31.656666 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:32.034350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:32.034247 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:32.617172 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:32.614601 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:33.285345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:33.285306 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:33.893011 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:33.892972 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:34.502186 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:34.502146 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:35.110967 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:35.110927 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:35.712782 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:35.712667 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:36.388491 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:36.388450 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:37.226012 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:37.225976 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:38.081042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:38.081004 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:38.270800 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:38.270747 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:38.434746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:38.434522 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:38.632715 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:38.632638 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:39.081736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:39.079883 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:39.495532 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:39.495488 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:39.784056 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:39.783963 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:39.987248 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:39.987212 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:40.375906 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:40.375866 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:40.610173 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:40.610133 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:40.753483 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:40.753436 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:40.907715 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:40.903810 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:41.050271 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:41.050174 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:41.549342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:41.549302 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:41.699112 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:41.699060 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:41.831313 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:41.831224 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:42.500956 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:42.500909 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:43.503795 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:43.503748 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:44.483721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:44.483660 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:44.661223 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:44.661180 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:45.165329 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:45.165290 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:45.316479 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:45.316437 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:45.675414 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:45.675374 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:45.835432 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:45.835387 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:45.987240 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:45.987146 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:46.170726 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:46.170688 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:46.321196 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:46.321098 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:46.971059 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:46.971021 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:47.178497 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:47.177419 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:47.434416 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:37:47.434347 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:37:48.060439 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:48.060395 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:48.206308 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:48.206274 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:48.454159 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:48.454125 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:48.699172 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:48.699023 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:48.848170 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:48.847781 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:49.010075 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:49.010039 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:49.240877 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:49.240832 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:49.881564 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:49.881523 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:50.518894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:50.518851 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:51.418866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:51.418822 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:52.156648 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:52.156583 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:52.777789 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:52.777748 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:53.258320 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:53.258278 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:53.402361 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:53.402326 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:53.662771 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:53.662188 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:54.404230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:54.404184 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:54.542329 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:54.542293 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:55.410303 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:55.410263 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:56.148052 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:56.148010 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:56.587181 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:56.587144 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:56.918021 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:56.917983 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:57.121779 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:57.121693 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:57.585951 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:57.585900 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:57.724122 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:57.724078 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:57.872653 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:57.872570 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:58.389509 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:58.389474 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:59.180492 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:59.180452 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:37:59.326199 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:37:59.323058 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:00.107867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:00.107823 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:00.243540 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:00.243506 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:00.625190 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:00.625156 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:00.776700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:00.767347 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:01.113098 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.113057 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf"] Oct 31 03:38:01.114300 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.114264 5490 cpu_manager.go:414] "containerMap: removing container" podUID="0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" containerName="istio-proxy" Oct 31 03:38:01.114300 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.114297 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" containerName="istio-proxy" Oct 31 03:38:01.114513 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.114377 5490 cpu_manager.go:414] "containerMap: removing container" podUID="5aa4a40b-944a-4113-9e1e-50c844f77cb4" containerName="istio-proxy" Oct 31 03:38:01.114513 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.114386 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa4a40b-944a-4113-9e1e-50c844f77cb4" containerName="istio-proxy" Oct 31 03:38:01.114638 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.114583 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa4a40b-944a-4113-9e1e-50c844f77cb4" containerName="istio-proxy" Oct 31 03:38:01.114728 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.114657 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e6d3eaf-534e-4de2-929e-e32cbb4d0ea9" containerName="istio-proxy" Oct 31 03:38:01.159007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.158970 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf"] Oct 31 03:38:01.159258 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.159176 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" Oct 31 03:38:01.176136 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.176062 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-d04aad-serving-cert\"" Oct 31 03:38:01.198212 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.198177 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c809aea3-9195-4f90-9ffb-ca153af1a5be-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf\" (UID: \"c809aea3-9195-4f90-9ffb-ca153af1a5be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" Oct 31 03:38:01.198532 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.198510 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c809aea3-9195-4f90-9ffb-ca153af1a5be-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf\" (UID: \"c809aea3-9195-4f90-9ffb-ca153af1a5be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" Oct 31 03:38:01.198721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.198702 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c809aea3-9195-4f90-9ffb-ca153af1a5be-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf\" (UID: \"c809aea3-9195-4f90-9ffb-ca153af1a5be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" Oct 31 03:38:01.198803 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.198761 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4nf2\" (UniqueName: \"kubernetes.io/projected/c809aea3-9195-4f90-9ffb-ca153af1a5be-kube-api-access-k4nf2\") pod \"maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf\" (UID: \"c809aea3-9195-4f90-9ffb-ca153af1a5be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" Oct 31 03:38:01.214766 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.214732 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:01.303193 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.303152 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c809aea3-9195-4f90-9ffb-ca153af1a5be-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf\" (UID: \"c809aea3-9195-4f90-9ffb-ca153af1a5be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" Oct 31 03:38:01.303811 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.303339 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c809aea3-9195-4f90-9ffb-ca153af1a5be-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf\" (UID: \"c809aea3-9195-4f90-9ffb-ca153af1a5be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" Oct 31 03:38:01.303811 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.303466 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c809aea3-9195-4f90-9ffb-ca153af1a5be-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf\" (UID: \"c809aea3-9195-4f90-9ffb-ca153af1a5be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" Oct 31 03:38:01.303811 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.303497 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4nf2\" (UniqueName: \"kubernetes.io/projected/c809aea3-9195-4f90-9ffb-ca153af1a5be-kube-api-access-k4nf2\") pod \"maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf\" (UID: \"c809aea3-9195-4f90-9ffb-ca153af1a5be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" Oct 31 03:38:01.303811 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:38:01.303772 5490 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-serving-cert: secret "maas-api-e2e-shared-b-w3-d04aad-serving-cert" not found Oct 31 03:38:01.304044 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:38:01.303843 5490 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c809aea3-9195-4f90-9ffb-ca153af1a5be-maas-api-tls podName:c809aea3-9195-4f90-9ffb-ca153af1a5be nodeName:}" failed. No retries permitted until 2026-10-31 03:38:01.803822449 +0000 UTC m=+1316.338517394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c809aea3-9195-4f90-9ffb-ca153af1a5be-maas-api-tls") pod "maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" (UID: "c809aea3-9195-4f90-9ffb-ca153af1a5be") : secret "maas-api-e2e-shared-b-w3-d04aad-serving-cert" not found Oct 31 03:38:01.304544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.304517 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c809aea3-9195-4f90-9ffb-ca153af1a5be-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf\" (UID: \"c809aea3-9195-4f90-9ffb-ca153af1a5be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" Oct 31 03:38:01.306180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.306156 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c809aea3-9195-4f90-9ffb-ca153af1a5be-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf\" (UID: \"c809aea3-9195-4f90-9ffb-ca153af1a5be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" Oct 31 03:38:01.312827 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.312793 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4nf2\" (UniqueName: \"kubernetes.io/projected/c809aea3-9195-4f90-9ffb-ca153af1a5be-kube-api-access-k4nf2\") pod \"maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf\" (UID: \"c809aea3-9195-4f90-9ffb-ca153af1a5be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" Oct 31 03:38:01.365062 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.364967 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:01.528369 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.528327 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:01.678862 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.678825 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:01.814094 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.814049 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c809aea3-9195-4f90-9ffb-ca153af1a5be-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf\" (UID: \"c809aea3-9195-4f90-9ffb-ca153af1a5be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" Oct 31 03:38:01.818009 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.817715 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c809aea3-9195-4f90-9ffb-ca153af1a5be-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf\" (UID: \"c809aea3-9195-4f90-9ffb-ca153af1a5be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" Oct 31 03:38:01.820479 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:01.820445 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:02.098052 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:02.097966 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" Oct 31 03:38:02.429573 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:02.429539 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf"] Oct 31 03:38:02.431443 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:38:02.431411 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc809aea3_9195_4f90_9ffb_ca153af1a5be.slice/crio-4435cc1da3eb892efdb59cbc95418ca8432d59c7de9ae87c7fb1934a1d5a0062 WatchSource:0}: Error finding container 4435cc1da3eb892efdb59cbc95418ca8432d59c7de9ae87c7fb1934a1d5a0062: Status 404 returned error can't find the container with id 4435cc1da3eb892efdb59cbc95418ca8432d59c7de9ae87c7fb1934a1d5a0062 Oct 31 03:38:02.492176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:02.492135 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:02.503282 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:02.503224 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" event={"ID":"c809aea3-9195-4f90-9ffb-ca153af1a5be","Type":"ContainerStarted","Data":"4435cc1da3eb892efdb59cbc95418ca8432d59c7de9ae87c7fb1934a1d5a0062"} Oct 31 03:38:02.884304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:02.884265 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:03.036704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:03.033349 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:03.513109 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:03.513074 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" event={"ID":"c809aea3-9195-4f90-9ffb-ca153af1a5be","Type":"ContainerStarted","Data":"6b5b8525fb106f3dcb186e908c507929a427d3baf29f5efb12f872eccc7b0ffd"} Oct 31 03:38:03.513514 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:03.513150 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" Oct 31 03:38:03.533927 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:03.533860 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" podStartSLOduration=2.100141447 podStartE2EDuration="2.533844809s" totalImagesPullingTime="433.703362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:03.53282678 +0000 UTC m=+1318.067521737" watchObservedRunningTime="2026-10-31 03:38:03.533844809 +0000 UTC m=+1318.068539765" Oct 31 03:38:03.658886 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:03.658848 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:04.577350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:04.577311 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:05.233543 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:05.233497 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:05.447814 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:05.447778 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:05.695103 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:05.695065 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:05.896290 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:05.896244 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:06.353465 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:06.353428 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:06.600363 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:06.595105 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:06.753438 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:06.753384 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:06.897340 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:06.895424 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:07.290697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:07.287462 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:08.003710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:07.993727 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:08.822164 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:08.814946 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:09.541242 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:09.541199 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" Oct 31 03:38:09.905140 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:09.904644 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7"] Oct 31 03:38:09.918013 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:09.917980 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" Oct 31 03:38:09.919548 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:09.919517 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7"] Oct 31 03:38:09.935698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:09.924454 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8385c5-serving-cert\"" Oct 31 03:38:09.990478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:09.990437 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/428d2eec-1622-4031-9390-905eb9ede375-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7\" (UID: \"428d2eec-1622-4031-9390-905eb9ede375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" Oct 31 03:38:09.990758 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:09.990733 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcb4j\" (UniqueName: \"kubernetes.io/projected/428d2eec-1622-4031-9390-905eb9ede375-kube-api-access-qcb4j\") pod \"maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7\" (UID: \"428d2eec-1622-4031-9390-905eb9ede375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" Oct 31 03:38:09.990981 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:09.990963 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428d2eec-1622-4031-9390-905eb9ede375-metrics-certs\") pod \"maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7\" (UID: \"428d2eec-1622-4031-9390-905eb9ede375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" Oct 31 03:38:09.991206 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:09.991186 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/428d2eec-1622-4031-9390-905eb9ede375-maas-api-tls\") pod \"maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7\" (UID: \"428d2eec-1622-4031-9390-905eb9ede375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" Oct 31 03:38:10.003912 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:10.003872 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:10.092381 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:10.092341 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/428d2eec-1622-4031-9390-905eb9ede375-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7\" (UID: \"428d2eec-1622-4031-9390-905eb9ede375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" Oct 31 03:38:10.092381 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:10.092384 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcb4j\" (UniqueName: \"kubernetes.io/projected/428d2eec-1622-4031-9390-905eb9ede375-kube-api-access-qcb4j\") pod \"maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7\" (UID: \"428d2eec-1622-4031-9390-905eb9ede375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" Oct 31 03:38:10.092696 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:10.092423 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428d2eec-1622-4031-9390-905eb9ede375-metrics-certs\") pod \"maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7\" (UID: \"428d2eec-1622-4031-9390-905eb9ede375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" Oct 31 03:38:10.092696 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:10.092517 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/428d2eec-1622-4031-9390-905eb9ede375-maas-api-tls\") pod \"maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7\" (UID: \"428d2eec-1622-4031-9390-905eb9ede375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" Oct 31 03:38:10.093200 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:10.093172 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/428d2eec-1622-4031-9390-905eb9ede375-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7\" (UID: \"428d2eec-1622-4031-9390-905eb9ede375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" Oct 31 03:38:10.098364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:10.095897 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/428d2eec-1622-4031-9390-905eb9ede375-maas-api-tls\") pod \"maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7\" (UID: \"428d2eec-1622-4031-9390-905eb9ede375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" Oct 31 03:38:10.098364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:10.098265 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428d2eec-1622-4031-9390-905eb9ede375-metrics-certs\") pod \"maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7\" (UID: \"428d2eec-1622-4031-9390-905eb9ede375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" Oct 31 03:38:10.120387 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:10.118731 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcb4j\" (UniqueName: \"kubernetes.io/projected/428d2eec-1622-4031-9390-905eb9ede375-kube-api-access-qcb4j\") pod \"maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7\" (UID: \"428d2eec-1622-4031-9390-905eb9ede375\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" Oct 31 03:38:10.154599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:10.154559 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:10.247782 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:10.247692 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" Oct 31 03:38:10.567760 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:10.564719 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7"] Oct 31 03:38:10.695772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:10.695736 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" event={"ID":"428d2eec-1622-4031-9390-905eb9ede375","Type":"ContainerStarted","Data":"f2ecc56201eff767d77a237ba0e44985cf614424f53a0ca3d096936c4cabfdda"} Oct 31 03:38:10.807880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:10.806771 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:11.535326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:11.535287 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:11.706570 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:11.706535 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" event={"ID":"428d2eec-1622-4031-9390-905eb9ede375","Type":"ContainerStarted","Data":"b4ad916de4d443291279095bf3dd4edfe09a55252f3da5fd8fdb8568a66e5883"} Oct 31 03:38:11.707059 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:11.706616 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" Oct 31 03:38:11.725528 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:11.725474 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" podStartSLOduration=2.301047984 podStartE2EDuration="2.725457523s" totalImagesPullingTime="424.409539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:11.723189246 +0000 UTC m=+1326.257884202" watchObservedRunningTime="2026-10-31 03:38:11.725457523 +0000 UTC m=+1326.260152477" Oct 31 03:38:12.345130 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:12.345088 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:12.999154 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:12.989657 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:13.538453 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:13.538418 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:13.674870 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:13.674830 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:13.904370 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:13.904304 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:14.464925 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:14.464883 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:15.393701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:15.390364 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:16.079908 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:16.079861 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:16.632699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:16.630311 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:16.933195 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:16.933148 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:17.131603 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:38:17.131569 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:38:17.674896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:17.674855 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:17.725364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:17.725330 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" Oct 31 03:38:18.585706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:18.585641 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:19.160334 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:19.160286 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:19.289728 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:19.289671 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:19.655607 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:19.655513 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:19.791905 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:19.791860 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:19.924375 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:19.924278 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:20.460477 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:20.460424 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:20.607179 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:20.607103 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:20.809435 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:20.809335 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:21.088298 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:21.088214 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:21.447389 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:21.447335 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:21.584806 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:21.584772 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:21.941567 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:21.932929 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:22.131818 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:22.131783 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:22.382954 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:22.382811 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:22.688572 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:22.688516 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:23.491409 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:23.491375 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:23.650607 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:23.650560 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:24.341700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:24.340886 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:24.707720 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:24.706604 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:24.846585 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:24.846547 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:25.268813 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:25.268773 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:26.138014 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:26.137962 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:26.350779 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:26.350741 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:26.538352 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:26.538309 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:26.933922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:26.933883 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:27.454911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:27.451389 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:27.653976 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:27.653944 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:27.686438 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:27.686403 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq"] Oct 31 03:38:27.691769 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:27.691748 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq" Oct 31 03:38:27.695567 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:27.695538 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-619d46\"" Oct 31 03:38:27.696891 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:27.696847 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-619d46-dockercfg-5w94j\"" Oct 31 03:38:27.700978 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:27.700955 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq"] Oct 31 03:38:27.741041 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:27.740953 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm89w\" (UniqueName: \"kubernetes.io/projected/0426e928-eaba-427f-956a-e344126aeb39-kube-api-access-nm89w\") pod \"payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq\" (UID: \"0426e928-eaba-427f-956a-e344126aeb39\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq" Oct 31 03:38:27.741041 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:27.741019 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0426e928-eaba-427f-956a-e344126aeb39-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq\" (UID: \"0426e928-eaba-427f-956a-e344126aeb39\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq" Oct 31 03:38:27.878703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:27.878154 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm89w\" (UniqueName: \"kubernetes.io/projected/0426e928-eaba-427f-956a-e344126aeb39-kube-api-access-nm89w\") pod \"payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq\" (UID: \"0426e928-eaba-427f-956a-e344126aeb39\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq" Oct 31 03:38:27.878703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:27.878216 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0426e928-eaba-427f-956a-e344126aeb39-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq\" (UID: \"0426e928-eaba-427f-956a-e344126aeb39\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq" Oct 31 03:38:27.887707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:27.887406 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0426e928-eaba-427f-956a-e344126aeb39-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq\" (UID: \"0426e928-eaba-427f-956a-e344126aeb39\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq" Oct 31 03:38:27.897112 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:27.897075 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:28.034985 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.034196 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm89w\" (UniqueName: \"kubernetes.io/projected/0426e928-eaba-427f-956a-e344126aeb39-kube-api-access-nm89w\") pod \"payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq\" (UID: \"0426e928-eaba-427f-956a-e344126aeb39\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq" Oct 31 03:38:28.130255 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.130193 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:28.321392 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.321308 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq" Oct 31 03:38:28.462519 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.462477 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:28.473532 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.473487 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd"] Oct 31 03:38:28.481370 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.481344 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.489236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.489204 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b2dc82-openshift-default-dockercfg-nr4wl\"" Oct 31 03:38:28.490830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.490425 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd"] Oct 31 03:38:28.617126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.603419 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:28.659703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.652463 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c1369b1-6901-4b11-a05d-12763e10bf4f-istiod-ca-cert\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.659703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.652523 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-envoy\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.659703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.652622 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-credential-socket\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.659703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.652657 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-workload-certs\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.659703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.652702 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-data\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.659703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.652883 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-workload-socket\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.659703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.653939 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4jmk\" (UniqueName: \"kubernetes.io/projected/1c1369b1-6901-4b11-a05d-12763e10bf4f-kube-api-access-h4jmk\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.659703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.654085 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-podinfo\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.659703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.654137 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-token\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.757599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.757043 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-podinfo\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.757599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.757119 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-token\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.757599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.757255 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c1369b1-6901-4b11-a05d-12763e10bf4f-istiod-ca-cert\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.757599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.757301 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-envoy\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.757599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.757397 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-credential-socket\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.757599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.757432 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-workload-certs\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.757599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.757462 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-data\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.757599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.757551 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-workload-socket\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.758292 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.757725 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4jmk\" (UniqueName: \"kubernetes.io/projected/1c1369b1-6901-4b11-a05d-12763e10bf4f-kube-api-access-h4jmk\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.758641 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.758612 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-data\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.758932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.758907 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-workload-socket\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.764647 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.762177 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-credential-socket\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.764647 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.762879 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c1369b1-6901-4b11-a05d-12763e10bf4f-istiod-ca-cert\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.764647 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.763317 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-workload-certs\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.764922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.764828 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq"] Oct 31 03:38:28.776301 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.776265 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-podinfo\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.776622 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.776581 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-envoy\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.792344 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.792309 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4jmk\" (UniqueName: \"kubernetes.io/projected/1c1369b1-6901-4b11-a05d-12763e10bf4f-kube-api-access-h4jmk\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.810760 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.803185 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-token\") pod \"e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:28.843956 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.843916 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:29.000530 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:28.996377 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq" event={"ID":"0426e928-eaba-427f-956a-e344126aeb39","Type":"ContainerStarted","Data":"559707ae13d15c0ca92d28269b05e853700458f47a7a7918bb4a7cd45a0ceef9"} Oct 31 03:38:29.231417 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:38:29.231371 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1369b1_6901_4b11_a05d_12763e10bf4f.slice/crio-3d40810aaff12b111fc3d32906c2d5bdc46dc522e65df4dd3a889e79e98c508e WatchSource:0}: Error finding container 3d40810aaff12b111fc3d32906c2d5bdc46dc522e65df4dd3a889e79e98c508e: Status 404 returned error can't find the container with id 3d40810aaff12b111fc3d32906c2d5bdc46dc522e65df4dd3a889e79e98c508e Oct 31 03:38:29.231576 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:29.231466 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd"] Oct 31 03:38:29.234967 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:29.234934 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:38:29.235753 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:29.235006 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:38:29.235753 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:29.235043 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:38:29.322905 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:29.322790 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:29.473650 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:29.473450 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:30.009368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:30.009315 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq" event={"ID":"0426e928-eaba-427f-956a-e344126aeb39","Type":"ContainerStarted","Data":"fd11a51073a4e5943aa4cbeec2843a719f5a3708d3454fc0d84ed04e6b0d2d89"} Oct 31 03:38:30.009762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:30.009739 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq" Oct 31 03:38:30.012209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:30.012176 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" event={"ID":"1c1369b1-6901-4b11-a05d-12763e10bf4f","Type":"ContainerStarted","Data":"3ac46324dd80274db98b0c7d9e5804c4075a0969b6cf7a5d5a172477c4e31216"} Oct 31 03:38:30.012351 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:30.012230 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" event={"ID":"1c1369b1-6901-4b11-a05d-12763e10bf4f","Type":"ContainerStarted","Data":"3d40810aaff12b111fc3d32906c2d5bdc46dc522e65df4dd3a889e79e98c508e"} Oct 31 03:38:30.033294 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:30.033237 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq" podStartSLOduration=3.033220448 podStartE2EDuration="3.033220448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:30.027371446 +0000 UTC m=+1344.562066413" watchObservedRunningTime="2026-10-31 03:38:30.033220448 +0000 UTC m=+1344.567915404" Oct 31 03:38:30.056444 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:30.056392 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" podStartSLOduration=2.056375436 podStartE2EDuration="2.056375436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:30.051110096 +0000 UTC m=+1344.585805051" watchObservedRunningTime="2026-10-31 03:38:30.056375436 +0000 UTC m=+1344.591070395" Oct 31 03:38:30.515022 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:30.514985 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:30.852175 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:30.852083 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:30.860594 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:30.860558 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:31.022137 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:31.022097 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:31.023949 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:31.023921 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:38:31.463132 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:31.463084 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:32.290750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:32.290276 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:32.441327 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:32.441291 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:33.231872 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:33.231822 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:33.579380 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:33.579292 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:33.765700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:33.761025 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:33.966702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:33.966545 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:34.512920 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:34.512880 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:34.648326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:34.648291 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:34.884009 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:34.883922 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q"] Oct 31 03:38:34.892527 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:34.892485 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:34.904318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:34.904154 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q"] Oct 31 03:38:34.910639 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:34.909892 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6f28b4-openshift-default-dockercfg-2jgls\"" Oct 31 03:38:34.965519 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:34.965474 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:35.011710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.008161 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-workload-certs\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.011710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.008340 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-istio-envoy\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.011710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.008422 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-credential-socket\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.011710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.008612 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82755388-258d-4de0-bdfe-59073fb3dfac-istiod-ca-cert\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.011710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.008657 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82755388-258d-4de0-bdfe-59073fb3dfac-istio-podinfo\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.011710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.009039 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-istio-data\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.011710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.009855 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sc8p\" (UniqueName: \"kubernetes.io/projected/82755388-258d-4de0-bdfe-59073fb3dfac-kube-api-access-6sc8p\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.011710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.009956 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-workload-socket\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.011710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.010097 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82755388-258d-4de0-bdfe-59073fb3dfac-istio-token\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.152761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.112866 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-workload-certs\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.152761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.112949 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-istio-envoy\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.152761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.112990 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-credential-socket\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.152761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.113077 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82755388-258d-4de0-bdfe-59073fb3dfac-istiod-ca-cert\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.152761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.113124 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82755388-258d-4de0-bdfe-59073fb3dfac-istio-podinfo\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.152761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.113261 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-istio-data\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.152761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.113353 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sc8p\" (UniqueName: \"kubernetes.io/projected/82755388-258d-4de0-bdfe-59073fb3dfac-kube-api-access-6sc8p\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.152761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.113381 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-workload-socket\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.152761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.113473 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82755388-258d-4de0-bdfe-59073fb3dfac-istio-token\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.152761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.114289 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-workload-certs\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.152761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.120770 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-istio-data\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.152761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.121374 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-workload-socket\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.152761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.123206 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-credential-socket\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.152761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.123429 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82755388-258d-4de0-bdfe-59073fb3dfac-istiod-ca-cert\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.152761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.140638 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sc8p\" (UniqueName: \"kubernetes.io/projected/82755388-258d-4de0-bdfe-59073fb3dfac-kube-api-access-6sc8p\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.152761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.143578 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-istio-envoy\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.189697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.185206 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82755388-258d-4de0-bdfe-59073fb3dfac-istio-token\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.208788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.207648 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82755388-258d-4de0-bdfe-59073fb3dfac-istio-podinfo\") pod \"e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.234701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.234135 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:35.548231 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.548185 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:35.639178 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.622313 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q"] Oct 31 03:38:35.639178 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:38:35.623927 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82755388_258d_4de0_bdfe_59073fb3dfac.slice/crio-faeb723318aa4ea89e9d5dc763d371a6a583b7df429e2aa11a9f7867f8fa6688 WatchSource:0}: Error finding container faeb723318aa4ea89e9d5dc763d371a6a583b7df429e2aa11a9f7867f8fa6688: Status 404 returned error can't find the container with id faeb723318aa4ea89e9d5dc763d371a6a583b7df429e2aa11a9f7867f8fa6688 Oct 31 03:38:35.639178 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.628070 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:38:35.639178 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.628196 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:38:35.639178 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.628265 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:38:35.678061 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:35.674973 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:36.182486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:36.182450 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" event={"ID":"82755388-258d-4de0-bdfe-59073fb3dfac","Type":"ContainerStarted","Data":"c561dcbe0b0d736ebfdbe624c225a88e00c975cb9d15a6e0d036cea2cfa0a9fa"} Oct 31 03:38:36.182757 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:36.182506 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" event={"ID":"82755388-258d-4de0-bdfe-59073fb3dfac","Type":"ContainerStarted","Data":"faeb723318aa4ea89e9d5dc763d371a6a583b7df429e2aa11a9f7867f8fa6688"} Oct 31 03:38:36.229435 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:36.217838 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" podStartSLOduration=2.217814523 podStartE2EDuration="2.217814523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:36.204531092 +0000 UTC m=+1350.739226048" watchObservedRunningTime="2026-10-31 03:38:36.217814523 +0000 UTC m=+1350.752509478" Oct 31 03:38:36.234600 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:36.234567 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:36.241045 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:36.241019 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:36.542228 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:36.542102 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:37.094153 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:37.090322 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:37.192124 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:37.192093 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:37.193104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:37.193076 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:38:37.973427 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:37.973381 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:38.680174 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:38.680136 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:38.894216 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:38.894180 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:39.541150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:39.541107 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:39.876650 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:39.876511 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:40.024281 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:40.024239 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:40.963174 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:40.963132 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:41.029351 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:41.028983 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq" Oct 31 03:38:42.096601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:42.096567 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:42.275245 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:42.275198 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:42.891609 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:42.890420 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:43.066286 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:43.066251 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:43.366888 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:43.366822 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:45.723112 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:45.723064 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:45.784027 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:45.783994 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq"] Oct 31 03:38:45.789233 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:45.789197 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" Oct 31 03:38:45.804593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:45.804557 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq"] Oct 31 03:38:45.815981 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:45.815949 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-619d46-serving-cert\"" Oct 31 03:38:45.838663 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:45.838608 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c4be15d-9315-4ec9-8f92-1d60da143716-maas-api-tls\") pod \"maas-api-e2e-api-b-619d46-cf84cb896-hmjdq\" (UID: \"2c4be15d-9315-4ec9-8f92-1d60da143716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" Oct 31 03:38:45.838884 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:45.838732 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4be15d-9315-4ec9-8f92-1d60da143716-metrics-certs\") pod \"maas-api-e2e-api-b-619d46-cf84cb896-hmjdq\" (UID: \"2c4be15d-9315-4ec9-8f92-1d60da143716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" Oct 31 03:38:45.838884 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:45.838796 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c4be15d-9315-4ec9-8f92-1d60da143716-openshift-service-ca\") pod \"maas-api-e2e-api-b-619d46-cf84cb896-hmjdq\" (UID: \"2c4be15d-9315-4ec9-8f92-1d60da143716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" Oct 31 03:38:45.839188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:45.839144 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvzll\" (UniqueName: \"kubernetes.io/projected/2c4be15d-9315-4ec9-8f92-1d60da143716-kube-api-access-nvzll\") pod \"maas-api-e2e-api-b-619d46-cf84cb896-hmjdq\" (UID: \"2c4be15d-9315-4ec9-8f92-1d60da143716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" Oct 31 03:38:45.869165 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:45.869053 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:45.940653 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:45.940589 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvzll\" (UniqueName: \"kubernetes.io/projected/2c4be15d-9315-4ec9-8f92-1d60da143716-kube-api-access-nvzll\") pod \"maas-api-e2e-api-b-619d46-cf84cb896-hmjdq\" (UID: \"2c4be15d-9315-4ec9-8f92-1d60da143716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" Oct 31 03:38:45.940933 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:45.940782 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c4be15d-9315-4ec9-8f92-1d60da143716-maas-api-tls\") pod \"maas-api-e2e-api-b-619d46-cf84cb896-hmjdq\" (UID: \"2c4be15d-9315-4ec9-8f92-1d60da143716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" Oct 31 03:38:45.940933 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:45.940885 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4be15d-9315-4ec9-8f92-1d60da143716-metrics-certs\") pod \"maas-api-e2e-api-b-619d46-cf84cb896-hmjdq\" (UID: \"2c4be15d-9315-4ec9-8f92-1d60da143716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" Oct 31 03:38:45.941065 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:45.940963 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c4be15d-9315-4ec9-8f92-1d60da143716-openshift-service-ca\") pod \"maas-api-e2e-api-b-619d46-cf84cb896-hmjdq\" (UID: \"2c4be15d-9315-4ec9-8f92-1d60da143716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" Oct 31 03:38:45.941752 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:45.941721 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c4be15d-9315-4ec9-8f92-1d60da143716-openshift-service-ca\") pod \"maas-api-e2e-api-b-619d46-cf84cb896-hmjdq\" (UID: \"2c4be15d-9315-4ec9-8f92-1d60da143716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" Oct 31 03:38:45.943415 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:45.943384 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c4be15d-9315-4ec9-8f92-1d60da143716-maas-api-tls\") pod \"maas-api-e2e-api-b-619d46-cf84cb896-hmjdq\" (UID: \"2c4be15d-9315-4ec9-8f92-1d60da143716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" Oct 31 03:38:45.943520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:45.943440 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4be15d-9315-4ec9-8f92-1d60da143716-metrics-certs\") pod \"maas-api-e2e-api-b-619d46-cf84cb896-hmjdq\" (UID: \"2c4be15d-9315-4ec9-8f92-1d60da143716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" Oct 31 03:38:45.951960 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:45.951934 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvzll\" (UniqueName: \"kubernetes.io/projected/2c4be15d-9315-4ec9-8f92-1d60da143716-kube-api-access-nvzll\") pod \"maas-api-e2e-api-b-619d46-cf84cb896-hmjdq\" (UID: \"2c4be15d-9315-4ec9-8f92-1d60da143716\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" Oct 31 03:38:46.000950 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:46.000856 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:46.137401 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:46.131697 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" Oct 31 03:38:46.143634 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:46.143587 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:46.477439 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:46.477406 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq"] Oct 31 03:38:46.626751 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:46.622018 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:46.706708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:46.705742 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" event={"ID":"2c4be15d-9315-4ec9-8f92-1d60da143716","Type":"ContainerStarted","Data":"79c03c453c0d69d1eebdc5643896313c53f65587c74a6ff90703cacda0371cfa"} Oct 31 03:38:47.088929 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:47.081762 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:47.148607 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:38:47.148491 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:38:47.280933 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:47.280809 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:47.448293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:47.448257 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:47.723566 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:47.723530 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" event={"ID":"2c4be15d-9315-4ec9-8f92-1d60da143716","Type":"ContainerStarted","Data":"2c272cce3f9954a076159c93d64182b34cd238a9f042def44dfb6640c82d4963"} Oct 31 03:38:47.723800 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:47.723602 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" Oct 31 03:38:47.786455 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:47.786396 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" podStartSLOduration=2.254911482 podStartE2EDuration="2.78638045s" totalImagesPullingTime="531.468968ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:47.76674212 +0000 UTC m=+1362.301437076" watchObservedRunningTime="2026-10-31 03:38:47.78638045 +0000 UTC m=+1362.321075405" Oct 31 03:38:47.841699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:47.841501 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:48.035755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:48.035157 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:48.626096 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:48.626054 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:49.315491 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:49.315235 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:49.474698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:49.471739 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:49.715700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:49.715553 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:50.623619 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:50.623574 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:51.904855 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:51.904805 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:52.712055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:52.712013 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:52.909948 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:52.909898 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:53.063754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:53.062135 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:53.742898 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:53.742866 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" Oct 31 03:38:54.079144 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:54.079060 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:54.241024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:54.240981 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:54.546701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:54.537652 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:55.088814 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.085739 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs"] Oct 31 03:38:55.093826 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.093800 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" Oct 31 03:38:55.130003 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.129972 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b2dc82-serving-cert\"" Oct 31 03:38:55.136811 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.136777 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs"] Oct 31 03:38:55.141174 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.141145 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-metrics-certs\") pod \"maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs\" (UID: \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" Oct 31 03:38:55.141352 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.141333 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs\" (UID: \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" Oct 31 03:38:55.141530 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.141474 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t875r\" (UniqueName: \"kubernetes.io/projected/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-kube-api-access-t875r\") pod \"maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs\" (UID: \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" Oct 31 03:38:55.141530 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.141545 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-maas-api-tls\") pod \"maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs\" (UID: \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" Oct 31 03:38:55.176256 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.176213 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:55.243357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.243321 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs\" (UID: \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" Oct 31 03:38:55.243512 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.243468 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t875r\" (UniqueName: \"kubernetes.io/projected/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-kube-api-access-t875r\") pod \"maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs\" (UID: \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" Oct 31 03:38:55.243588 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.243530 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-maas-api-tls\") pod \"maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs\" (UID: \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" Oct 31 03:38:55.243642 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.243598 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-metrics-certs\") pod \"maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs\" (UID: \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" Oct 31 03:38:55.243734 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:38:55.243657 5490 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-serving-cert: secret "maas-api-e2e-inf-a-b2dc82-serving-cert" not found Oct 31 03:38:55.243816 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:38:55.243803 5490 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-maas-api-tls podName:2cf3d600-6bdb-46fe-857b-219e0c3ba93c nodeName:}" failed. No retries permitted until 2026-10-31 03:38:55.743780702 +0000 UTC m=+1370.278475650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-maas-api-tls") pod "maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" (UID: "2cf3d600-6bdb-46fe-857b-219e0c3ba93c") : secret "maas-api-e2e-inf-a-b2dc82-serving-cert" not found Oct 31 03:38:55.244072 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.244051 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs\" (UID: \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" Oct 31 03:38:55.246563 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.246534 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-metrics-certs\") pod \"maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs\" (UID: \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" Oct 31 03:38:55.253442 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.253409 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t875r\" (UniqueName: \"kubernetes.io/projected/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-kube-api-access-t875r\") pod \"maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs\" (UID: \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" Oct 31 03:38:55.310810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.310762 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:55.604340 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.604301 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:55.751929 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.751888 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-maas-api-tls\") pod \"maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs\" (UID: \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" Oct 31 03:38:55.754455 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.754421 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-maas-api-tls\") pod \"maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs\" (UID: \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" Oct 31 03:38:55.756262 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:55.756241 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" Oct 31 03:38:56.063122 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:56.063081 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:56.148531 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:38:56.148494 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf3d600_6bdb_46fe_857b_219e0c3ba93c.slice/crio-0b18eaabaaf99a2c3f17970cc9230578b9681d0942f6677e241a1ba14d87661a WatchSource:0}: Error finding container 0b18eaabaaf99a2c3f17970cc9230578b9681d0942f6677e241a1ba14d87661a: Status 404 returned error can't find the container with id 0b18eaabaaf99a2c3f17970cc9230578b9681d0942f6677e241a1ba14d87661a Oct 31 03:38:56.156307 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:56.156249 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs"] Oct 31 03:38:56.855997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:56.855901 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" event={"ID":"2cf3d600-6bdb-46fe-857b-219e0c3ba93c","Type":"ContainerStarted","Data":"54b4f4bb363ff26938fbc292eec3023772a87e303ab6884494598dce69f0ac81"} Oct 31 03:38:56.855997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:56.855954 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" event={"ID":"2cf3d600-6bdb-46fe-857b-219e0c3ba93c","Type":"ContainerStarted","Data":"0b18eaabaaf99a2c3f17970cc9230578b9681d0942f6677e241a1ba14d87661a"} Oct 31 03:38:56.856195 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:56.856029 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" Oct 31 03:38:56.876865 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:56.876804 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" podStartSLOduration=1.474826661 podStartE2EDuration="1.876785749s" totalImagesPullingTime="401.959088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:56.871581158 +0000 UTC m=+1371.406276130" watchObservedRunningTime="2026-10-31 03:38:56.876785749 +0000 UTC m=+1371.411480706" Oct 31 03:38:56.966310 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:56.966269 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:57.513126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:57.513090 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r"] Oct 31 03:38:57.513992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:57.513959 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" podUID="1efde673-8a62-4eb0-b2cd-3fad992e1867" containerName="maas-api" containerID="cri-o://b3e8db7d3f0c06d19c253e0b08a6131bb0777971c39f28e6fb329bdada2b0c25" gracePeriod=30 Oct 31 03:38:57.600995 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:57.600955 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:57.730535 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:57.730433 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:57.826837 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:57.826812 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" Oct 31 03:38:57.871779 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:57.871741 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:57.873577 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:57.873548 5490 generic.go:356] "Generic (PLEG): container finished" podID="1efde673-8a62-4eb0-b2cd-3fad992e1867" containerID="b3e8db7d3f0c06d19c253e0b08a6131bb0777971c39f28e6fb329bdada2b0c25" exitCode=0 Oct 31 03:38:57.873895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:57.873862 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" Oct 31 03:38:57.874010 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:57.873984 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" event={"ID":"1efde673-8a62-4eb0-b2cd-3fad992e1867","Type":"ContainerDied","Data":"b3e8db7d3f0c06d19c253e0b08a6131bb0777971c39f28e6fb329bdada2b0c25"} Oct 31 03:38:57.874777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:57.874118 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r" event={"ID":"1efde673-8a62-4eb0-b2cd-3fad992e1867","Type":"ContainerDied","Data":"dfd8b6ee4ada11f43d48fb8b939f2c716cb10961deeaa837a713cffa32833760"} Oct 31 03:38:57.874777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:57.874197 5490 scope.go:122] "RemoveContainer" containerID="b3e8db7d3f0c06d19c253e0b08a6131bb0777971c39f28e6fb329bdada2b0c25" Oct 31 03:38:57.924705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:57.920818 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1efde673-8a62-4eb0-b2cd-3fad992e1867-kube-api-access-xb84b\" (UniqueName: \"kubernetes.io/projected/1efde673-8a62-4eb0-b2cd-3fad992e1867-kube-api-access-xb84b\") pod \"1efde673-8a62-4eb0-b2cd-3fad992e1867\" (UID: \"1efde673-8a62-4eb0-b2cd-3fad992e1867\") " Oct 31 03:38:57.924705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:57.920911 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1efde673-8a62-4eb0-b2cd-3fad992e1867-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efde673-8a62-4eb0-b2cd-3fad992e1867-openshift-service-ca\") pod \"1efde673-8a62-4eb0-b2cd-3fad992e1867\" (UID: \"1efde673-8a62-4eb0-b2cd-3fad992e1867\") " Oct 31 03:38:57.924705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:57.921079 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1efde673-8a62-4eb0-b2cd-3fad992e1867-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1efde673-8a62-4eb0-b2cd-3fad992e1867-maas-api-tls\") pod \"1efde673-8a62-4eb0-b2cd-3fad992e1867\" (UID: \"1efde673-8a62-4eb0-b2cd-3fad992e1867\") " Oct 31 03:38:57.924705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:57.921135 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1efde673-8a62-4eb0-b2cd-3fad992e1867-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1efde673-8a62-4eb0-b2cd-3fad992e1867-metrics-certs\") pod \"1efde673-8a62-4eb0-b2cd-3fad992e1867\" (UID: \"1efde673-8a62-4eb0-b2cd-3fad992e1867\") " Oct 31 03:38:57.965942 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:57.965897 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1efde673-8a62-4eb0-b2cd-3fad992e1867-openshift-service-ca" pod "1efde673-8a62-4eb0-b2cd-3fad992e1867" (UID: "1efde673-8a62-4eb0-b2cd-3fad992e1867"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:38:58.031797 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:58.026559 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1efde673-8a62-4eb0-b2cd-3fad992e1867-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:58.111876 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:58.111841 5490 scope.go:122] "RemoveContainer" containerID="b3e8db7d3f0c06d19c253e0b08a6131bb0777971c39f28e6fb329bdada2b0c25" Oct 31 03:38:58.112521 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:38:58.112490 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3e8db7d3f0c06d19c253e0b08a6131bb0777971c39f28e6fb329bdada2b0c25\": container with ID starting with b3e8db7d3f0c06d19c253e0b08a6131bb0777971c39f28e6fb329bdada2b0c25 not found: ID does not exist" containerID="b3e8db7d3f0c06d19c253e0b08a6131bb0777971c39f28e6fb329bdada2b0c25" Oct 31 03:38:58.112657 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:58.112630 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e8db7d3f0c06d19c253e0b08a6131bb0777971c39f28e6fb329bdada2b0c25"} err="failed to get container status \"b3e8db7d3f0c06d19c253e0b08a6131bb0777971c39f28e6fb329bdada2b0c25\": rpc error: code = NotFound desc = could not find container \"b3e8db7d3f0c06d19c253e0b08a6131bb0777971c39f28e6fb329bdada2b0c25\": container with ID starting with b3e8db7d3f0c06d19c253e0b08a6131bb0777971c39f28e6fb329bdada2b0c25 not found: ID does not exist" Oct 31 03:38:58.119445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:58.117514 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1efde673-8a62-4eb0-b2cd-3fad992e1867-maas-api-tls" pod "1efde673-8a62-4eb0-b2cd-3fad992e1867" (UID: "1efde673-8a62-4eb0-b2cd-3fad992e1867"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:38:58.119445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:58.117865 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1efde673-8a62-4eb0-b2cd-3fad992e1867-kube-api-access-xb84b" pod "1efde673-8a62-4eb0-b2cd-3fad992e1867" (UID: "1efde673-8a62-4eb0-b2cd-3fad992e1867"). InnerVolumeSpecName "kube-api-access-xb84b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:38:58.121534 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:58.121509 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1efde673-8a62-4eb0-b2cd-3fad992e1867-metrics-certs" pod "1efde673-8a62-4eb0-b2cd-3fad992e1867" (UID: "1efde673-8a62-4eb0-b2cd-3fad992e1867"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:38:58.129368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:58.129283 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1efde673-8a62-4eb0-b2cd-3fad992e1867-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:58.129368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:58.129330 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1efde673-8a62-4eb0-b2cd-3fad992e1867-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:58.129368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:58.129345 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb84b\" (UniqueName: \"kubernetes.io/projected/1efde673-8a62-4eb0-b2cd-3fad992e1867-kube-api-access-xb84b\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:38:58.219908 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:58.219869 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r"] Oct 31 03:38:58.227392 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:58.227351 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b87769-649bdd7ff5-q758r"] Oct 31 03:38:58.319195 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:58.319133 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:59.217232 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:59.217190 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:38:59.589981 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:38:59.589890 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1efde673-8a62-4eb0-b2cd-3fad992e1867" path="/var/lib/kubelet/pods/1efde673-8a62-4eb0-b2cd-3fad992e1867/volumes" Oct 31 03:39:00.111343 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:00.111301 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:00.689100 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:00.688378 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:01.321477 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:01.321422 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:01.801624 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:01.801580 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s"] Oct 31 03:39:01.802661 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:01.802634 5490 cpu_manager.go:414] "containerMap: removing container" podUID="1efde673-8a62-4eb0-b2cd-3fad992e1867" containerName="maas-api" Oct 31 03:39:01.802826 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:01.802661 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1efde673-8a62-4eb0-b2cd-3fad992e1867" containerName="maas-api" Oct 31 03:39:01.802928 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:01.802913 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="1efde673-8a62-4eb0-b2cd-3fad992e1867" containerName="maas-api" Oct 31 03:39:01.809212 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:01.809191 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" Oct 31 03:39:01.812385 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:01.812364 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6f28b4-serving-cert\"" Oct 31 03:39:01.815318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:01.815292 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s"] Oct 31 03:39:01.903507 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:01.903465 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-maas-api-tls\") pod \"maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s\" (UID: \"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" Oct 31 03:39:01.903687 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:01.903590 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-metrics-certs\") pod \"maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s\" (UID: \"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" Oct 31 03:39:01.903815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:01.903795 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s\" (UID: \"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" Oct 31 03:39:01.903857 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:01.903835 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlkw7\" (UniqueName: \"kubernetes.io/projected/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-kube-api-access-zlkw7\") pod \"maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s\" (UID: \"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" Oct 31 03:39:01.951293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:01.951257 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:02.005506 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:02.005467 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s\" (UID: \"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" Oct 31 03:39:02.005770 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:02.005520 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlkw7\" (UniqueName: \"kubernetes.io/projected/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-kube-api-access-zlkw7\") pod \"maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s\" (UID: \"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" Oct 31 03:39:02.005770 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:02.005737 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-maas-api-tls\") pod \"maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s\" (UID: \"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" Oct 31 03:39:02.005922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:02.005792 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-metrics-certs\") pod \"maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s\" (UID: \"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" Oct 31 03:39:02.006205 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:02.006178 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s\" (UID: \"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" Oct 31 03:39:02.008236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:02.008211 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-metrics-certs\") pod \"maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s\" (UID: \"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" Oct 31 03:39:02.008414 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:02.008396 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-maas-api-tls\") pod \"maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s\" (UID: \"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" Oct 31 03:39:02.013360 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:02.013331 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlkw7\" (UniqueName: \"kubernetes.io/projected/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-kube-api-access-zlkw7\") pod \"maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s\" (UID: \"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" Oct 31 03:39:02.082298 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:02.082207 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:02.129548 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:02.129505 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" Oct 31 03:39:02.447508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:02.447477 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s"] Oct 31 03:39:02.450285 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:39:02.450253 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3cbb24_ff7d_4913_ad1e_f7bd39470ce9.slice/crio-66a3e4e635ebc7dbd44da427a83f3a4b4b047822fef20af9be57ff37959a4efe WatchSource:0}: Error finding container 66a3e4e635ebc7dbd44da427a83f3a4b4b047822fef20af9be57ff37959a4efe: Status 404 returned error can't find the container with id 66a3e4e635ebc7dbd44da427a83f3a4b4b047822fef20af9be57ff37959a4efe Oct 31 03:39:02.688587 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:02.688549 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:02.881137 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:02.881069 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" Oct 31 03:39:02.927483 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:02.927444 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" event={"ID":"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9","Type":"ContainerStarted","Data":"66a3e4e635ebc7dbd44da427a83f3a4b4b047822fef20af9be57ff37959a4efe"} Oct 31 03:39:03.549431 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.549392 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:03.658602 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.658566 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf"] Oct 31 03:39:03.663065 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.663038 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.670094 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.670071 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8385c5-openshift-default-dockercfg-vs5dp\"" Oct 31 03:39:03.699170 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.699121 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf"] Oct 31 03:39:03.704526 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.704491 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5"] Oct 31 03:39:03.710467 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.710441 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.739746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.739655 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5"] Oct 31 03:39:03.760320 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.760289 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-credential-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.760507 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.760394 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-token\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.760577 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.760525 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-podinfo\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.760577 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.760560 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggvln\" (UniqueName: \"kubernetes.io/projected/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-kube-api-access-ggvln\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.760843 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.760594 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-workload-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.760843 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.760646 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-workload-certs\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.760843 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.760775 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-envoy\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.760843 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.760830 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istiod-ca-cert\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.761453 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.760939 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-data\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.862809 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.862672 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-envoy\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.862988 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.862804 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-podinfo\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.862988 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.862880 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-podinfo\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.862988 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.862931 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggvln\" (UniqueName: \"kubernetes.io/projected/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-kube-api-access-ggvln\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.862988 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.862961 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-workload-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.863137 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.863012 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-workload-certs\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.863137 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.863108 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-envoy\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.863240 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.863141 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-workload-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.863240 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.863192 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istiod-ca-cert\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.863355 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.863325 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-workload-certs\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.863451 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.863428 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlbln\" (UniqueName: \"kubernetes.io/projected/240a8cbb-b982-4490-9929-ece58bc06d5f-kube-api-access-nlbln\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.863521 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.863478 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-data\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.863583 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.863515 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-workload-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.863583 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.863546 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-workload-certs\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.863760 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.863619 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/240a8cbb-b982-4490-9929-ece58bc06d5f-istiod-ca-cert\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.863760 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.863705 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-credential-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.863885 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.863800 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-credential-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.863885 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.863841 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-data\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.863998 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.863921 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-token\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.863998 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.863958 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-data\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.863998 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.863963 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istiod-ca-cert\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.863998 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.863988 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-token\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.864218 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.864137 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-credential-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.865575 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.865550 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-envoy\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.865834 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.865815 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-podinfo\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.871829 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.871808 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-token\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.872238 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.872222 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggvln\" (UniqueName: \"kubernetes.io/projected/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-kube-api-access-ggvln\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:03.937552 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.937512 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" event={"ID":"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9","Type":"ContainerStarted","Data":"394975da730c6e79753f6d5a355214ac961a6878bda14c5520ab0c651a0086d3"} Oct 31 03:39:03.937977 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.937760 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" Oct 31 03:39:03.961987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.961933 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" podStartSLOduration=2.529244457 podStartE2EDuration="2.961918185s" totalImagesPullingTime="432.673728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:03.960636539 +0000 UTC m=+1378.495331495" watchObservedRunningTime="2026-10-31 03:39:03.961918185 +0000 UTC m=+1378.496613176" Oct 31 03:39:03.966245 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.966212 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-workload-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.966419 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.966284 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-workload-certs\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.966419 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.966315 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlbln\" (UniqueName: \"kubernetes.io/projected/240a8cbb-b982-4490-9929-ece58bc06d5f-kube-api-access-nlbln\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.966419 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.966374 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/240a8cbb-b982-4490-9929-ece58bc06d5f-istiod-ca-cert\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.966419 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.966405 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-credential-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.966665 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.966517 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-data\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.966665 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.966548 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-token\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.966665 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.966614 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-envoy\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.966665 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.966650 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-podinfo\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.966925 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.966754 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-workload-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.966925 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.966765 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-workload-certs\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.966925 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.966910 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-credential-socket\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.967083 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.966941 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-data\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.967135 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.967115 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/240a8cbb-b982-4490-9929-ece58bc06d5f-istiod-ca-cert\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.969206 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.969162 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-envoy\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.969360 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.969342 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-podinfo\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.979003 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.978973 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-token\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.979113 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.979052 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlbln\" (UniqueName: \"kubernetes.io/projected/240a8cbb-b982-4490-9929-ece58bc06d5f-kube-api-access-nlbln\") pod \"e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:03.982059 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:03.982039 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:04.054942 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:04.054898 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:04.278156 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:39:04.276859 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459ac1eb_d4bf_4b60_badd_07fe1893f3d0.slice/crio-5eef03c392f67ee66d7961e4ee9f1b00072643747de17bb20a2c0b6ba652929a WatchSource:0}: Error finding container 5eef03c392f67ee66d7961e4ee9f1b00072643747de17bb20a2c0b6ba652929a: Status 404 returned error can't find the container with id 5eef03c392f67ee66d7961e4ee9f1b00072643747de17bb20a2c0b6ba652929a Oct 31 03:39:04.286964 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:04.286933 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf"] Oct 31 03:39:04.289086 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:04.289047 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:39:04.289198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:04.289137 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:39:04.289198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:04.289183 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:39:04.444100 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:04.444067 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5"] Oct 31 03:39:04.445344 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:39:04.445317 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240a8cbb_b982_4490_9929_ece58bc06d5f.slice/crio-c162e37fa7494025341a427887ab3a93e447d57984db4f1ddc99fe4736d823bb WatchSource:0}: Error finding container c162e37fa7494025341a427887ab3a93e447d57984db4f1ddc99fe4736d823bb: Status 404 returned error can't find the container with id c162e37fa7494025341a427887ab3a93e447d57984db4f1ddc99fe4736d823bb Oct 31 03:39:04.449038 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:04.449004 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:39:04.449129 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:04.449081 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:39:04.449129 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:04.449112 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:39:04.488834 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:04.488163 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:04.949259 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:04.949208 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" event={"ID":"459ac1eb-d4bf-4b60-badd-07fe1893f3d0","Type":"ContainerStarted","Data":"21b58edfaf6c7c047e9abe559094ad809945cd1aea40f442e2c758eb6ea9efd8"} Oct 31 03:39:04.949725 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:04.949281 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" event={"ID":"459ac1eb-d4bf-4b60-badd-07fe1893f3d0","Type":"ContainerStarted","Data":"5eef03c392f67ee66d7961e4ee9f1b00072643747de17bb20a2c0b6ba652929a"} Oct 31 03:39:04.952407 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:04.952358 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" event={"ID":"240a8cbb-b982-4490-9929-ece58bc06d5f","Type":"ContainerStarted","Data":"6f51f90454ddb1cfcb3b2ca63d7d4a468c72144e80f5b07c30f553266611e729"} Oct 31 03:39:04.952784 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:04.952428 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" event={"ID":"240a8cbb-b982-4490-9929-ece58bc06d5f","Type":"ContainerStarted","Data":"c162e37fa7494025341a427887ab3a93e447d57984db4f1ddc99fe4736d823bb"} Oct 31 03:39:04.980235 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:04.980180 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" podStartSLOduration=1.980164099 podStartE2EDuration="1.980164099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:04.973724577 +0000 UTC m=+1379.508419533" watchObservedRunningTime="2026-10-31 03:39:04.980164099 +0000 UTC m=+1379.514859116" Oct 31 03:39:04.982764 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:04.982729 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:04.997160 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:04.997097 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" podStartSLOduration=1.99707824 podStartE2EDuration="1.99707824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:04.994401265 +0000 UTC m=+1379.529096221" watchObservedRunningTime="2026-10-31 03:39:04.99707824 +0000 UTC m=+1379.531773195" Oct 31 03:39:05.055175 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:05.055144 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:05.057317 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:05.057284 5490 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 03:39:05.057478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:05.057348 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" podUID="240a8cbb-b982-4490-9929-ece58bc06d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 03:39:05.305661 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:05.305548 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:05.489704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:05.486166 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:06.302690 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:06.273384 5490 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 03:39:06.401574 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:06.401528 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" podUID="240a8cbb-b982-4490-9929-ece58bc06d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 03:39:06.524697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:06.363744 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:06.524697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:06.279964 5490 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:39:06.524697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:06.517451 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" podUID="459ac1eb-d4bf-4b60-badd-07fe1893f3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 03:39:06.681010 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:06.680617 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:06.817167 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:06.817133 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:07.022719 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:07.022071 5490 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:39:07.022719 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:07.022139 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" podUID="459ac1eb-d4bf-4b60-badd-07fe1893f3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 03:39:07.104870 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:07.104829 5490 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 03:39:07.105694 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:07.105636 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" podUID="240a8cbb-b982-4490-9929-ece58bc06d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 03:39:07.211345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:07.211301 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:07.448204 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:07.448162 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:07.991735 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:07.991647 5490 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:39:07.991952 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:07.991765 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" podUID="459ac1eb-d4bf-4b60-badd-07fe1893f3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 03:39:08.062711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:08.061474 5490 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 03:39:08.062711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:08.061544 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" podUID="240a8cbb-b982-4490-9929-ece58bc06d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 03:39:08.224270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:08.224229 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:08.982722 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:08.982637 5490 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:39:08.983295 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:08.983262 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" podUID="459ac1eb-d4bf-4b60-badd-07fe1893f3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 03:39:08.993448 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:08.993405 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:09.057710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:09.056291 5490 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 03:39:09.057710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:09.056355 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" podUID="240a8cbb-b982-4490-9929-ece58bc06d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 03:39:09.984139 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:09.982655 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" Oct 31 03:39:09.984139 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:09.983193 5490 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:39:09.984139 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:09.983248 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" podUID="459ac1eb-d4bf-4b60-badd-07fe1893f3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 03:39:10.019126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:10.019078 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:10.084703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:10.084191 5490 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 03:39:10.084703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:10.084259 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" podUID="240a8cbb-b982-4490-9929-ece58bc06d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 03:39:10.946201 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:10.946164 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:10.985809 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:10.985772 5490 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:39:10.986979 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:10.986942 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" podUID="459ac1eb-d4bf-4b60-badd-07fe1893f3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 03:39:11.120702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:11.107540 5490 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 03:39:11.120702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:11.107612 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" podUID="240a8cbb-b982-4490-9929-ece58bc06d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 03:39:12.030742 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:12.026000 5490 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 03:39:12.030742 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:12.026080 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" podUID="459ac1eb-d4bf-4b60-badd-07fe1893f3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 03:39:12.069632 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:12.056158 5490 patch_prober.go:29] interesting pod/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 03:39:12.069632 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:12.056222 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" podUID="240a8cbb-b982-4490-9929-ece58bc06d5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 03:39:12.080104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:12.079922 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:12.264824 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:12.264783 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:12.406774 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:12.406035 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:13.000441 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:13.000403 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:13.002576 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:13.002527 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:13.006413 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:13.006369 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:39:13.061882 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:13.061846 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:13.063526 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:13.063480 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:13.064543 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:13.064522 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:39:13.515415 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:13.515384 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:14.497424 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:14.497301 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:15.885218 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:15.885178 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:16.351328 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:16.351272 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:16.511256 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:16.511204 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:16.731395 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:16.731354 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:17.112805 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:17.105225 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:17.140778 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:39:17.140741 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:39:17.444539 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:17.444492 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:17.713441 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:17.713338 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:17.945729 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:17.943256 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:18.697462 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:18.697392 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:18.897914 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:18.897879 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:19.033198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:19.033093 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:19.686608 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:19.686564 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:20.676097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:20.676057 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:21.346310 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:21.346249 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:22.091458 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:22.091421 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:23.289204 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:23.289155 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:23.432327 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:23.432279 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:23.841372 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:23.841334 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:24.322500 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:24.322453 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:24.914698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:24.893695 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:25.604968 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:25.604917 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:26.014213 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:26.014175 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:26.898894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:26.898809 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:27.564428 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:27.560553 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:28.291749 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:28.291713 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:28.864578 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:28.864536 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:28.998760 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:28.998724 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:29.265768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:29.265725 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:29.447733 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:29.447634 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:29.686696 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:29.686570 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:30.443747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:30.443700 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:31.449263 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:31.449222 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:32.064866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:32.064814 5490 patch_prober.go:29] interesting pod/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:39:32.065101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:32.064898 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" podUID="1c1369b1-6901-4b11-a05d-12763e10bf4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:39:32.201474 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:32.201435 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:32.885073 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:32.885033 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:33.581013 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:33.580919 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:34.171199 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:34.171157 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:34.773736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:34.773659 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:34.981412 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:34.981358 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:35.945217 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:35.945142 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:36.376853 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:36.376753 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:36.742462 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:36.742414 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:37.507913 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:37.507874 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:38.274701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:38.272715 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:38.771099 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:38.771061 5490 generic.go:356] "Generic (PLEG): container finished" podID="1c1369b1-6901-4b11-a05d-12763e10bf4f" containerID="3ac46324dd80274db98b0c7d9e5804c4075a0969b6cf7a5d5a172477c4e31216" exitCode=0 Oct 31 03:39:38.771578 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:38.771147 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" event={"ID":"1c1369b1-6901-4b11-a05d-12763e10bf4f","Type":"ContainerDied","Data":"3ac46324dd80274db98b0c7d9e5804c4075a0969b6cf7a5d5a172477c4e31216"} Oct 31 03:39:38.771703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:38.771659 5490 scope.go:122] "RemoveContainer" containerID="3ac46324dd80274db98b0c7d9e5804c4075a0969b6cf7a5d5a172477c4e31216" Oct 31 03:39:38.774338 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:38.774304 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:39:38.774444 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:38.774400 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:39:38.774509 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:38.774459 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:39:38.904345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:38.904220 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:39.614642 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:39.614565 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:39.766699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:39.762071 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:39.797551 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:39.797508 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" event={"ID":"1c1369b1-6901-4b11-a05d-12763e10bf4f","Type":"ContainerStarted","Data":"e012428bb113329314fc4e7cd260cd3464f5146ed6816ff9d9b6685de55ec84b"} Oct 31 03:39:39.856210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:39.856173 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:39:39.856426 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:39.856289 5490 patch_prober.go:29] interesting pod/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 03:39:39.856426 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:39.856335 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" podUID="1c1369b1-6901-4b11-a05d-12763e10bf4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 03:39:40.519413 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:40.519170 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:40.852257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:40.852157 5490 patch_prober.go:29] interesting pod/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 03:39:40.852257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:40.852233 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" podUID="1c1369b1-6901-4b11-a05d-12763e10bf4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 03:39:41.623043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:41.622943 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:41.852632 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:41.852575 5490 patch_prober.go:29] interesting pod/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 03:39:41.853157 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:41.852640 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" podUID="1c1369b1-6901-4b11-a05d-12763e10bf4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 03:39:42.486298 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:42.484248 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:42.852639 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:42.852554 5490 patch_prober.go:29] interesting pod/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 03:39:42.852639 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:42.852614 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" podUID="1c1369b1-6901-4b11-a05d-12763e10bf4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 03:39:43.096695 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:43.096639 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:43.855707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:43.854806 5490 patch_prober.go:29] interesting pod/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 03:39:43.855707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:43.854888 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" podUID="1c1369b1-6901-4b11-a05d-12763e10bf4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 03:39:43.977187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:43.977130 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:44.786876 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:44.786815 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:44.883701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:44.882859 5490 patch_prober.go:29] interesting pod/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Oct 31 03:39:44.883701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:44.882924 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" podUID="1c1369b1-6901-4b11-a05d-12763e10bf4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Oct 31 03:39:45.356282 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:45.356238 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:45.670345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:45.670309 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:45.857804 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:45.857771 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:39:45.858279 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:45.858261 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:39:45.859007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:45.858989 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:39:46.406524 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:46.406488 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:47.107950 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:47.107912 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:47.155971 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:39:47.155933 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:39:47.984099 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:47.983915 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:48.456736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:48.456653 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:48.757945 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:48.757850 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:49.466192 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:49.466148 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:50.176500 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:50.176458 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:50.870033 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:50.869993 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:51.505170 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:51.505137 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:51.664230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:51.664196 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:51.895760 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:51.878278 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:52.171915 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:52.171876 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:52.855806 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:52.855765 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:53.582080 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:53.582043 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:54.198441 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:54.198402 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:55.055380 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:55.055330 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:55.578226 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:55.578192 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:55.730367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:55.730306 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:55.971504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:55.971470 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:56.785018 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:56.784979 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:56.920284 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:56.920245 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:57.056489 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:57.056386 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:57.735908 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:57.724188 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:58.448985 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:58.448949 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:59.027610 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:59.027568 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:59.186432 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:59.186387 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:59.326397 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:59.326294 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:39:59.476251 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:39:59.476209 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:00.381248 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:00.381138 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:01.165347 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:01.165311 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:01.813867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:01.813819 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:02.418722 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:02.418631 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:03.285584 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:03.285547 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:03.881981 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:03.881773 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:04.814016 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:04.813978 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:05.943700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:05.936068 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:07.122032 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:07.121997 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:07.910606 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:07.910561 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:08.778519 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:08.778476 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:09.425190 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:09.425152 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:11.016656 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:11.016615 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:11.454710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:11.453669 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:12.013570 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:12.013519 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:12.199708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:12.199247 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:12.437620 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:12.437577 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:12.698996 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:12.698900 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:12.869303 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:12.869259 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:13.937365 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:13.937329 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:14.621593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:14.621547 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:15.340700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:15.326067 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:16.079201 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:16.071456 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:16.749422 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:16.749381 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:16.884528 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:16.884489 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:17.131104 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:40:17.131018 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:40:17.488103 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:17.488064 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:17.641390 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:17.641335 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:18.391233 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:18.391192 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:18.986921 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:18.986876 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:19.676753 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:19.676718 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:19.827841 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:19.827800 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:20.313849 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:20.313714 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:20.872505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:20.872346 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:21.755300 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:21.755044 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:22.398128 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:22.398079 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:23.055449 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:23.051476 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:23.235971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:23.235931 5490 patch_prober.go:29] interesting pod/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:40:23.236180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:23.236017 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" podUID="82755388-258d-4de0-bdfe-59073fb3dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:40:24.074803 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:24.074763 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:24.884393 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:24.884337 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:25.540749 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:25.530789 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:26.403451 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:26.403419 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:27.201626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:27.201552 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:28.018486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:28.018436 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:28.153192 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:28.153106 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:28.478498 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:28.478464 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:28.754997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:28.754915 5490 generic.go:356] "Generic (PLEG): container finished" podID="82755388-258d-4de0-bdfe-59073fb3dfac" containerID="c561dcbe0b0d736ebfdbe624c225a88e00c975cb9d15a6e0d036cea2cfa0a9fa" exitCode=0 Oct 31 03:40:28.754997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:28.754986 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" event={"ID":"82755388-258d-4de0-bdfe-59073fb3dfac","Type":"ContainerDied","Data":"c561dcbe0b0d736ebfdbe624c225a88e00c975cb9d15a6e0d036cea2cfa0a9fa"} Oct 31 03:40:28.755504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:28.755488 5490 scope.go:122] "RemoveContainer" containerID="c561dcbe0b0d736ebfdbe624c225a88e00c975cb9d15a6e0d036cea2cfa0a9fa" Oct 31 03:40:28.757810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:28.757778 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:40:28.757949 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:28.757855 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:40:28.757949 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:28.757922 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:40:28.826331 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:28.826293 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:29.178742 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:29.176037 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:29.765531 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:29.765495 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" event={"ID":"82755388-258d-4de0-bdfe-59073fb3dfac","Type":"ContainerStarted","Data":"966af05b50546a53479e02ae0dfcda86bc45800ba2292905b9052c8064c8673f"} Oct 31 03:40:29.800518 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:29.800478 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:29.941769 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:29.941734 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:30.234966 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:30.234931 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:40:30.235334 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:30.235311 5490 patch_prober.go:29] interesting pod/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 03:40:30.236430 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:30.236389 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" podUID="82755388-258d-4de0-bdfe-59073fb3dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 03:40:30.537981 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:30.537878 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:30.763736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:30.762365 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:31.249821 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:31.249776 5490 patch_prober.go:29] interesting pod/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 03:40:31.250442 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:31.250406 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" podUID="82755388-258d-4de0-bdfe-59073fb3dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 03:40:31.715437 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:31.715400 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq"] Oct 31 03:40:31.860721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:31.860641 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:32.004490 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:32.004339 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:32.171701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:32.171065 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:32.248699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:32.245925 5490 patch_prober.go:29] interesting pod/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 03:40:32.248699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:32.245995 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" podUID="82755388-258d-4de0-bdfe-59073fb3dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 03:40:33.237703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:33.236145 5490 patch_prober.go:29] interesting pod/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 03:40:33.237703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:33.236222 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" podUID="82755388-258d-4de0-bdfe-59073fb3dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 03:40:33.862354 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:33.862301 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:34.239173 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:34.239130 5490 patch_prober.go:29] interesting pod/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 03:40:34.239855 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:34.239819 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" podUID="82755388-258d-4de0-bdfe-59073fb3dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 03:40:35.249700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:35.249150 5490 patch_prober.go:29] interesting pod/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 31 03:40:35.249700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:35.249225 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" podUID="82755388-258d-4de0-bdfe-59073fb3dfac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 31 03:40:35.298304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:35.297949 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:36.240053 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:36.240020 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:40:36.240550 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:36.240531 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:40:36.241456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:36.241435 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:40:36.455073 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:36.455033 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:36.716582 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:36.716544 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq" podUID="0426e928-eaba-427f-956a-e344126aeb39" containerName="payload-pre-processing" containerID="cri-o://fd11a51073a4e5943aa4cbeec2843a719f5a3708d3454fc0d84ed04e6b0d2d89" gracePeriod=25 Oct 31 03:40:36.953705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:36.952807 5490 generic.go:356] "Generic (PLEG): container finished" podID="0426e928-eaba-427f-956a-e344126aeb39" containerID="fd11a51073a4e5943aa4cbeec2843a719f5a3708d3454fc0d84ed04e6b0d2d89" exitCode=0 Oct 31 03:40:36.954574 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:36.954540 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq" event={"ID":"0426e928-eaba-427f-956a-e344126aeb39","Type":"ContainerDied","Data":"fd11a51073a4e5943aa4cbeec2843a719f5a3708d3454fc0d84ed04e6b0d2d89"} Oct 31 03:40:37.066156 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:37.066116 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq" Oct 31 03:40:37.229164 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:37.229128 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0426e928-eaba-427f-956a-e344126aeb39-kube-api-access-nm89w\" (UniqueName: \"kubernetes.io/projected/0426e928-eaba-427f-956a-e344126aeb39-kube-api-access-nm89w\") pod \"0426e928-eaba-427f-956a-e344126aeb39\" (UID: \"0426e928-eaba-427f-956a-e344126aeb39\") " Oct 31 03:40:37.229509 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:37.229482 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0426e928-eaba-427f-956a-e344126aeb39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0426e928-eaba-427f-956a-e344126aeb39-plugins-config-volume\") pod \"0426e928-eaba-427f-956a-e344126aeb39\" (UID: \"0426e928-eaba-427f-956a-e344126aeb39\") " Oct 31 03:40:37.229960 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:37.229927 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0426e928-eaba-427f-956a-e344126aeb39-plugins-config-volume" pod "0426e928-eaba-427f-956a-e344126aeb39" (UID: "0426e928-eaba-427f-956a-e344126aeb39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:37.230521 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:37.230498 5490 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0426e928-eaba-427f-956a-e344126aeb39-plugins-config-volume\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:37.232013 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:37.231985 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0426e928-eaba-427f-956a-e344126aeb39-kube-api-access-nm89w" pod "0426e928-eaba-427f-956a-e344126aeb39" (UID: "0426e928-eaba-427f-956a-e344126aeb39"). InnerVolumeSpecName "kube-api-access-nm89w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:37.337544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:37.337434 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm89w\" (UniqueName: \"kubernetes.io/projected/0426e928-eaba-427f-956a-e344126aeb39-kube-api-access-nm89w\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:37.397236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:37.397190 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:37.965923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:37.965881 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq" Oct 31 03:40:37.966490 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:37.966449 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq" event={"ID":"0426e928-eaba-427f-956a-e344126aeb39","Type":"ContainerDied","Data":"559707ae13d15c0ca92d28269b05e853700458f47a7a7918bb4a7cd45a0ceef9"} Oct 31 03:40:37.966650 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:37.966533 5490 scope.go:122] "RemoveContainer" containerID="fd11a51073a4e5943aa4cbeec2843a719f5a3708d3454fc0d84ed04e6b0d2d89" Oct 31 03:40:38.002363 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:38.000963 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq"] Oct 31 03:40:38.008064 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:38.008020 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-619d46-5969574455-lpwjq"] Oct 31 03:40:38.337440 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:38.337348 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:39.580737 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:39.580696 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0426e928-eaba-427f-956a-e344126aeb39" path="/var/lib/kubelet/pods/0426e928-eaba-427f-956a-e344126aeb39/volumes" Oct 31 03:40:40.032652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:40.029530 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:41.046645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:41.046602 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:41.249971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:41.249922 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:41.439633 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:41.439595 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:41.673814 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:41.673776 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:42.349142 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:42.347233 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:42.614908 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:42.614785 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:42.652145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:42.652107 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz"] Oct 31 03:40:42.653027 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:42.653009 5490 cpu_manager.go:414] "containerMap: removing container" podUID="0426e928-eaba-427f-956a-e344126aeb39" containerName="payload-pre-processing" Oct 31 03:40:42.653097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:42.653029 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0426e928-eaba-427f-956a-e344126aeb39" containerName="payload-pre-processing" Oct 31 03:40:42.653174 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:42.653164 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="0426e928-eaba-427f-956a-e344126aeb39" containerName="payload-pre-processing" Oct 31 03:40:42.658886 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:42.658864 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz" Oct 31 03:40:42.669311 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:42.669275 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-73e222\"" Oct 31 03:40:42.669311 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:42.669302 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-73e222-dockercfg-6x22b\"" Oct 31 03:40:42.674732 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:42.674637 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz"] Oct 31 03:40:42.738742 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:42.738652 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc03805-5508-419e-84a2-e6d8494833f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz\" (UID: \"dfc03805-5508-419e-84a2-e6d8494833f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz" Oct 31 03:40:42.739068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:42.738845 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpd6f\" (UniqueName: \"kubernetes.io/projected/dfc03805-5508-419e-84a2-e6d8494833f7-kube-api-access-hpd6f\") pod \"payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz\" (UID: \"dfc03805-5508-419e-84a2-e6d8494833f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz" Oct 31 03:40:42.804516 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:42.804476 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:42.864701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:42.842171 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpd6f\" (UniqueName: \"kubernetes.io/projected/dfc03805-5508-419e-84a2-e6d8494833f7-kube-api-access-hpd6f\") pod \"payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz\" (UID: \"dfc03805-5508-419e-84a2-e6d8494833f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz" Oct 31 03:40:42.865560 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:42.865478 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc03805-5508-419e-84a2-e6d8494833f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz\" (UID: \"dfc03805-5508-419e-84a2-e6d8494833f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz" Oct 31 03:40:42.866880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:42.866850 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc03805-5508-419e-84a2-e6d8494833f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz\" (UID: \"dfc03805-5508-419e-84a2-e6d8494833f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz" Oct 31 03:40:42.923753 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:42.922363 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpd6f\" (UniqueName: \"kubernetes.io/projected/dfc03805-5508-419e-84a2-e6d8494833f7-kube-api-access-hpd6f\") pod \"payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz\" (UID: \"dfc03805-5508-419e-84a2-e6d8494833f7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz" Oct 31 03:40:42.989698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:42.985270 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz" Oct 31 03:40:43.520789 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:43.517576 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz"] Oct 31 03:40:43.755727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:43.755686 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:44.153296 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:44.153252 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz" event={"ID":"dfc03805-5508-419e-84a2-e6d8494833f7","Type":"ContainerStarted","Data":"19e11d7521d7ce41daf5566961ab97015c1783ff0d41d4755088f0547f1827a1"} Oct 31 03:40:44.153534 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:44.153505 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz" Oct 31 03:40:44.153598 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:44.153551 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz" event={"ID":"dfc03805-5508-419e-84a2-e6d8494833f7","Type":"ContainerStarted","Data":"099d0749c62fcaf412709fbda93c670af5c2d9676cb3106fa5b80f0de2d486ac"} Oct 31 03:40:44.179797 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:44.179734 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz" podStartSLOduration=2.17971366 podStartE2EDuration="2.17971366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:44.170952209 +0000 UTC m=+1478.705647167" watchObservedRunningTime="2026-10-31 03:40:44.17971366 +0000 UTC m=+1478.714408616" Oct 31 03:40:44.740370 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:44.740264 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:45.481705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:45.481648 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:46.185669 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:46.185620 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:46.551433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:46.551320 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:46.714377 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:46.714337 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:46.868757 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:46.868285 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:47.030700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:47.029978 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:47.230324 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:40:47.229924 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:40:47.290170 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:47.285110 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:47.702334 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:47.702297 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:48.637263 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:48.637184 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:49.459222 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:49.459179 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:49.687062 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:49.686971 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:50.186609 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:50.186492 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:50.655481 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:50.655445 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d"] Oct 31 03:40:50.655703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:50.655665 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" podUID="092826cc-3459-4bd9-b0f8-272ca8e44bf9" containerName="istio-proxy" containerID="cri-o://8037bc405e75074c52647dc92cd935eb862fae882a28ccf157fa249b74b223ea" gracePeriod=30 Oct 31 03:40:50.822093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:50.821851 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:51.703161 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:51.702614 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:52.293419 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:52.293377 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:52.909909 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:52.909792 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:53.758806 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:53.758745 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:54.335039 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:54.334998 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:54.474364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:54.474325 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:54.622442 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:54.622353 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:54.844703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:54.830757 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:55.165942 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:55.165915 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz" Oct 31 03:40:55.527484 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:55.527389 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:55.724169 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:55.724125 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:55.989220 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:55.989187 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:40:55.991181 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:55.991151 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:56.187741 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.187697 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh"] Oct 31 03:40:56.189375 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.189319 5490 cpu_manager.go:414] "containerMap: removing container" podUID="092826cc-3459-4bd9-b0f8-272ca8e44bf9" containerName="istio-proxy" Oct 31 03:40:56.189565 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.189550 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="092826cc-3459-4bd9-b0f8-272ca8e44bf9" containerName="istio-proxy" Oct 31 03:40:56.190001 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.189982 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="092826cc-3459-4bd9-b0f8-272ca8e44bf9" containerName="istio-proxy" Oct 31 03:40:56.418697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.415283 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.418697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.415806 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh"] Oct 31 03:40:56.424011 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.419707 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a5d9b6-openshift-default-dockercfg-fjkqj\"" Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.426054 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istiod-ca-cert\") pod \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.426149 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-workload-certs\") pod \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.426276 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-data\") pod \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.426341 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-workload-socket\") pod \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.426489 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-token\" (UniqueName: \"kubernetes.io/projected/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-token\") pod \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.426583 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-podinfo\") pod \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.426723 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-envoy\") pod \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.426852 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/092826cc-3459-4bd9-b0f8-272ca8e44bf9-kube-api-access-t7qnp\" (UniqueName: \"kubernetes.io/projected/092826cc-3459-4bd9-b0f8-272ca8e44bf9-kube-api-access-t7qnp\") pod \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.426926 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-credential-socket\") pod \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\" (UID: \"092826cc-3459-4bd9-b0f8-272ca8e44bf9\") " Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.427342 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.427397 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.427496 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.427520 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.427548 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.427601 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.427872 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.431303 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-workload-socket" pod "092826cc-3459-4bd9-b0f8-272ca8e44bf9" (UID: "092826cc-3459-4bd9-b0f8-272ca8e44bf9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.431617 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istiod-ca-cert" pod "092826cc-3459-4bd9-b0f8-272ca8e44bf9" (UID: "092826cc-3459-4bd9-b0f8-272ca8e44bf9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.431914 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-workload-certs" pod "092826cc-3459-4bd9-b0f8-272ca8e44bf9" (UID: "092826cc-3459-4bd9-b0f8-272ca8e44bf9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.432115 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-data" pod "092826cc-3459-4bd9-b0f8-272ca8e44bf9" (UID: "092826cc-3459-4bd9-b0f8-272ca8e44bf9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:40:56.434126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.432292 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-credential-socket" pod "092826cc-3459-4bd9-b0f8-272ca8e44bf9" (UID: "092826cc-3459-4bd9-b0f8-272ca8e44bf9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:40:56.437710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.437656 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prpt4\" (UniqueName: \"kubernetes.io/projected/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-kube-api-access-prpt4\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.437970 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.437919 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.439282 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.439257 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:56.439482 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.439462 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:56.439592 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.439580 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:56.439671 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.439661 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:56.439805 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.439779 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:56.446928 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.446891 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-envoy" pod "092826cc-3459-4bd9-b0f8-272ca8e44bf9" (UID: "092826cc-3459-4bd9-b0f8-272ca8e44bf9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:40:56.452707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.448324 5490 generic.go:356] "Generic (PLEG): container finished" podID="092826cc-3459-4bd9-b0f8-272ca8e44bf9" containerID="8037bc405e75074c52647dc92cd935eb862fae882a28ccf157fa249b74b223ea" exitCode=0 Oct 31 03:40:56.452707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.448425 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" Oct 31 03:40:56.452707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.448355 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" event={"ID":"092826cc-3459-4bd9-b0f8-272ca8e44bf9","Type":"ContainerDied","Data":"8037bc405e75074c52647dc92cd935eb862fae882a28ccf157fa249b74b223ea"} Oct 31 03:40:56.452707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.448876 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d" event={"ID":"092826cc-3459-4bd9-b0f8-272ca8e44bf9","Type":"ContainerDied","Data":"bc4b491932f97b2c77cb209237c9fe99dd96aa83a9f820af0eaf271ee0a3a4ea"} Oct 31 03:40:56.452707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.448912 5490 scope.go:122] "RemoveContainer" containerID="8037bc405e75074c52647dc92cd935eb862fae882a28ccf157fa249b74b223ea" Oct 31 03:40:56.491914 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.491849 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-podinfo" pod "092826cc-3459-4bd9-b0f8-272ca8e44bf9" (UID: "092826cc-3459-4bd9-b0f8-272ca8e44bf9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:40:56.491914 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.491907 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092826cc-3459-4bd9-b0f8-272ca8e44bf9-kube-api-access-t7qnp" pod "092826cc-3459-4bd9-b0f8-272ca8e44bf9" (UID: "092826cc-3459-4bd9-b0f8-272ca8e44bf9"). InnerVolumeSpecName "kube-api-access-t7qnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:56.492163 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.491951 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-token" pod "092826cc-3459-4bd9-b0f8-272ca8e44bf9" (UID: "092826cc-3459-4bd9-b0f8-272ca8e44bf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:40:56.511136 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.511099 5490 scope.go:122] "RemoveContainer" containerID="8037bc405e75074c52647dc92cd935eb862fae882a28ccf157fa249b74b223ea" Oct 31 03:40:56.511464 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:40:56.511441 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8037bc405e75074c52647dc92cd935eb862fae882a28ccf157fa249b74b223ea\": container with ID starting with 8037bc405e75074c52647dc92cd935eb862fae882a28ccf157fa249b74b223ea not found: ID does not exist" containerID="8037bc405e75074c52647dc92cd935eb862fae882a28ccf157fa249b74b223ea" Oct 31 03:40:56.511517 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.511475 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8037bc405e75074c52647dc92cd935eb862fae882a28ccf157fa249b74b223ea"} err="failed to get container status \"8037bc405e75074c52647dc92cd935eb862fae882a28ccf157fa249b74b223ea\": rpc error: code = NotFound desc = could not find container \"8037bc405e75074c52647dc92cd935eb862fae882a28ccf157fa249b74b223ea\": container with ID starting with 8037bc405e75074c52647dc92cd935eb862fae882a28ccf157fa249b74b223ea not found: ID does not exist" Oct 31 03:40:56.540851 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.540806 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.541023 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.540842 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.541023 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.540904 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.541023 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.540951 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.541236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.541144 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.541337 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.541317 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.541408 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.541362 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prpt4\" (UniqueName: \"kubernetes.io/projected/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-kube-api-access-prpt4\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.541465 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.541398 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.541465 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.541412 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.541623 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.541593 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.541799 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.541663 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.541854 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.541826 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:56.541854 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.541839 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:56.541935 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.541856 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092826cc-3459-4bd9-b0f8-272ca8e44bf9-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:56.541935 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.541868 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7qnp\" (UniqueName: \"kubernetes.io/projected/092826cc-3459-4bd9-b0f8-272ca8e44bf9-kube-api-access-t7qnp\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:40:56.542389 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.542364 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.542740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.542715 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.542834 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.542786 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.544938 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.544917 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.545021 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.544918 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.549055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.549028 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prpt4\" (UniqueName: \"kubernetes.io/projected/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-kube-api-access-prpt4\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.550513 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.550283 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.622457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.622415 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:56.762574 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.762530 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:56.782239 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.782198 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d"] Oct 31 03:40:56.785395 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.785360 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:56.788293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:56.788255 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-84h6d"] Oct 31 03:40:57.104985 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:40:57.104953 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2dbe7ba_df3f_4b3f_a6c8_6e386172702b.slice/crio-664c32076431a9e2b8e694813314df0fb14e1d9f435827bf764ac2dccabed58b WatchSource:0}: Error finding container 664c32076431a9e2b8e694813314df0fb14e1d9f435827bf764ac2dccabed58b: Status 404 returned error can't find the container with id 664c32076431a9e2b8e694813314df0fb14e1d9f435827bf764ac2dccabed58b Oct 31 03:40:57.105333 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:57.105034 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh"] Oct 31 03:40:57.108805 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:57.108766 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:40:57.108918 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:57.108847 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:40:57.108918 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:57.108876 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:40:57.395775 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:57.387612 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:57.464496 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:57.464458 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" event={"ID":"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b","Type":"ContainerStarted","Data":"1d426b91989be27ab5889cf4a028ee30b26e35646ac0bb25bbc43a9986204d2c"} Oct 31 03:40:57.464692 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:57.464522 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" event={"ID":"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b","Type":"ContainerStarted","Data":"664c32076431a9e2b8e694813314df0fb14e1d9f435827bf764ac2dccabed58b"} Oct 31 03:40:57.510397 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:57.507464 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" podStartSLOduration=2.5074386239999997 podStartE2EDuration="2.507438624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:57.489338667 +0000 UTC m=+1492.024033624" watchObservedRunningTime="2026-10-31 03:40:57.507438624 +0000 UTC m=+1492.042133586" Oct 31 03:40:57.581404 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:57.581362 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="092826cc-3459-4bd9-b0f8-272ca8e44bf9" path="/var/lib/kubelet/pods/092826cc-3459-4bd9-b0f8-272ca8e44bf9/volumes" Oct 31 03:40:57.785509 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:57.785471 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:58.075178 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:58.075094 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:58.792871 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:58.792838 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:58.873542 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:58.873503 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:59.483471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:59.483439 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:59.484510 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:59.484488 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:40:59.834464 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:59.834373 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:40:59.978329 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:40:59.978292 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:01.000557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:00.991055 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:01.313588 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:01.311727 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:01.773217 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:01.772658 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:02.301225 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:02.301181 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:03.159002 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:03.158945 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:03.343171 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:03.343128 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:03.518157 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:03.518071 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:04.183264 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:04.183226 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:05.075910 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:05.075870 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:05.687301 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:05.687265 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:06.323968 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:06.323925 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:06.857025 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:06.856987 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:07.034702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:07.029406 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:07.205383 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:07.205234 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:07.512284 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:07.512191 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:07.664037 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:07.663995 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:07.812326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:07.812169 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:08.567078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:08.567041 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq"] Oct 31 03:41:08.567524 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:08.567247 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" podUID="2c4be15d-9315-4ec9-8f92-1d60da143716" containerName="maas-api" containerID="cri-o://2c272cce3f9954a076159c93d64182b34cd238a9f042def44dfb6640c82d4963" gracePeriod=30 Oct 31 03:41:08.617299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:08.617250 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:08.736122 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:08.736079 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" podUID="2c4be15d-9315-4ec9-8f92-1d60da143716" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.113:8443/health\": dial tcp 10.133.0.113:8443: connect: connection refused" Oct 31 03:41:09.001042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:08.989274 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:09.233194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:09.233162 5490 generic.go:356] "Generic (PLEG): container finished" podID="2c4be15d-9315-4ec9-8f92-1d60da143716" containerID="2c272cce3f9954a076159c93d64182b34cd238a9f042def44dfb6640c82d4963" exitCode=0 Oct 31 03:41:09.236987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:09.233454 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" event={"ID":"2c4be15d-9315-4ec9-8f92-1d60da143716","Type":"ContainerDied","Data":"2c272cce3f9954a076159c93d64182b34cd238a9f042def44dfb6640c82d4963"} Oct 31 03:41:09.250749 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:09.250633 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:09.425648 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:09.425618 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" Oct 31 03:41:09.549624 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:09.549588 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c4be15d-9315-4ec9-8f92-1d60da143716-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c4be15d-9315-4ec9-8f92-1d60da143716-openshift-service-ca\") pod \"2c4be15d-9315-4ec9-8f92-1d60da143716\" (UID: \"2c4be15d-9315-4ec9-8f92-1d60da143716\") " Oct 31 03:41:09.549847 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:09.549729 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c4be15d-9315-4ec9-8f92-1d60da143716-kube-api-access-nvzll\" (UniqueName: \"kubernetes.io/projected/2c4be15d-9315-4ec9-8f92-1d60da143716-kube-api-access-nvzll\") pod \"2c4be15d-9315-4ec9-8f92-1d60da143716\" (UID: \"2c4be15d-9315-4ec9-8f92-1d60da143716\") " Oct 31 03:41:09.549950 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:09.549932 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c4be15d-9315-4ec9-8f92-1d60da143716-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c4be15d-9315-4ec9-8f92-1d60da143716-maas-api-tls\") pod \"2c4be15d-9315-4ec9-8f92-1d60da143716\" (UID: \"2c4be15d-9315-4ec9-8f92-1d60da143716\") " Oct 31 03:41:09.550015 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:09.549977 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c4be15d-9315-4ec9-8f92-1d60da143716-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4be15d-9315-4ec9-8f92-1d60da143716-metrics-certs\") pod \"2c4be15d-9315-4ec9-8f92-1d60da143716\" (UID: \"2c4be15d-9315-4ec9-8f92-1d60da143716\") " Oct 31 03:41:09.550068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:09.550011 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c4be15d-9315-4ec9-8f92-1d60da143716-openshift-service-ca" pod "2c4be15d-9315-4ec9-8f92-1d60da143716" (UID: "2c4be15d-9315-4ec9-8f92-1d60da143716"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:41:09.550908 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:09.550881 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c4be15d-9315-4ec9-8f92-1d60da143716-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:09.552161 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:09.552132 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4be15d-9315-4ec9-8f92-1d60da143716-kube-api-access-nvzll" pod "2c4be15d-9315-4ec9-8f92-1d60da143716" (UID: "2c4be15d-9315-4ec9-8f92-1d60da143716"). InnerVolumeSpecName "kube-api-access-nvzll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:09.552251 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:09.552169 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c4be15d-9315-4ec9-8f92-1d60da143716-maas-api-tls" pod "2c4be15d-9315-4ec9-8f92-1d60da143716" (UID: "2c4be15d-9315-4ec9-8f92-1d60da143716"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:41:09.552592 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:09.552561 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c4be15d-9315-4ec9-8f92-1d60da143716-metrics-certs" pod "2c4be15d-9315-4ec9-8f92-1d60da143716" (UID: "2c4be15d-9315-4ec9-8f92-1d60da143716"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:41:09.653372 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:09.653213 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:09.653958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:09.653817 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c4be15d-9315-4ec9-8f92-1d60da143716-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:09.653958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:09.653845 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c4be15d-9315-4ec9-8f92-1d60da143716-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:09.653958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:09.653862 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvzll\" (UniqueName: \"kubernetes.io/projected/2c4be15d-9315-4ec9-8f92-1d60da143716-kube-api-access-nvzll\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:09.915194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:09.915103 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:10.246349 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:10.246262 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" Oct 31 03:41:10.246531 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:10.246265 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq" event={"ID":"2c4be15d-9315-4ec9-8f92-1d60da143716","Type":"ContainerDied","Data":"79c03c453c0d69d1eebdc5643896313c53f65587c74a6ff90703cacda0371cfa"} Oct 31 03:41:10.246531 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:10.246424 5490 scope.go:122] "RemoveContainer" containerID="2c272cce3f9954a076159c93d64182b34cd238a9f042def44dfb6640c82d4963" Oct 31 03:41:10.284313 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:10.284272 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq"] Oct 31 03:41:10.302937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:10.293484 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-619d46-cf84cb896-hmjdq"] Oct 31 03:41:10.455272 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:10.449685 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:11.252081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:11.252028 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:11.602529 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:11.602421 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c4be15d-9315-4ec9-8f92-1d60da143716" path="/var/lib/kubelet/pods/2c4be15d-9315-4ec9-8f92-1d60da143716/volumes" Oct 31 03:41:11.619324 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:11.618049 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:12.442520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:12.442463 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:13.266219 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:13.266148 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:14.479602 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:14.479569 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:15.322740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:15.320922 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:16.092478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:16.092438 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:16.399795 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:16.399119 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:16.640709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:16.633647 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:17.112659 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:17.112606 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:17.132220 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:41:17.132181 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:41:17.550221 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:17.550176 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:18.497831 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:18.497794 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:19.672208 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:19.672165 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:19.959531 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:19.959441 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:20.493275 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:20.493233 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:21.306015 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:21.305971 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:21.585131 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:21.585043 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2"] Oct 31 03:41:21.585947 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:21.585922 5490 cpu_manager.go:414] "containerMap: removing container" podUID="2c4be15d-9315-4ec9-8f92-1d60da143716" containerName="maas-api" Oct 31 03:41:21.585947 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:21.585944 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c4be15d-9315-4ec9-8f92-1d60da143716" containerName="maas-api" Oct 31 03:41:21.586131 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:21.586118 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c4be15d-9315-4ec9-8f92-1d60da143716" containerName="maas-api" Oct 31 03:41:21.591529 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:21.591505 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:21.736726 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:21.728971 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-034fe8-openshift-default-dockercfg-vbg8p\"" Oct 31 03:41:21.794704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:21.790822 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:21.875784 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:21.872516 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:21.900871 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:21.897128 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2"] Oct 31 03:41:21.952698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:21.905990 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:21.952698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:21.906156 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:21.952698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:21.906195 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:21.952698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:21.906269 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:21.952698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:21.906450 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:21.952698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:21.906507 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:21.952698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:21.906599 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:21.952698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:21.906651 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsqtm\" (UniqueName: \"kubernetes.io/projected/cfa0b7ad-1a4c-4803-8214-e61e9f362694-kube-api-access-vsqtm\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:22.018697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.009116 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:22.067553 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.050265 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:22.067553 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.050372 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:22.067553 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.050490 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:22.067553 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.050554 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:22.067553 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.050653 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:22.067553 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.050711 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqtm\" (UniqueName: \"kubernetes.io/projected/cfa0b7ad-1a4c-4803-8214-e61e9f362694-kube-api-access-vsqtm\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:22.067553 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.050950 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:22.067553 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.051095 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:22.067553 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.051610 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:22.067553 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.021123 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:22.102777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.081321 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:22.174428 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.084890 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:22.174428 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.085135 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:22.174428 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.108797 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:22.191488 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.191452 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r"] Oct 31 03:41:22.191992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.191938 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" podUID="60771f19-56cc-40a1-a27c-b09f23ac9903" containerName="istio-proxy" containerID="cri-o://78c8fca1454169c1ff41d6bc7cb33be6114c658e21bbdff35bacfc579817faa1" gracePeriod=30 Oct 31 03:41:22.193894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.193791 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsqtm\" (UniqueName: \"kubernetes.io/projected/cfa0b7ad-1a4c-4803-8214-e61e9f362694-kube-api-access-vsqtm\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:22.194586 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.194362 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:22.195266 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.195237 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:22.210243 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.210202 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:22.219171 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.219139 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:22.544575 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.544545 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2"] Oct 31 03:41:22.546641 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:41:22.546607 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa0b7ad_1a4c_4803_8214_e61e9f362694.slice/crio-5bf06c04d4e177fec8166eaa38cc5726fcb8ce27ccf44d0d0f950455dd0f85d5 WatchSource:0}: Error finding container 5bf06c04d4e177fec8166eaa38cc5726fcb8ce27ccf44d0d0f950455dd0f85d5: Status 404 returned error can't find the container with id 5bf06c04d4e177fec8166eaa38cc5726fcb8ce27ccf44d0d0f950455dd0f85d5 Oct 31 03:41:22.550171 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.550135 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:41:22.550263 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.550204 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:41:22.550263 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:22.550233 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:41:23.355179 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:23.355144 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:23.482900 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:23.482858 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" event={"ID":"cfa0b7ad-1a4c-4803-8214-e61e9f362694","Type":"ContainerStarted","Data":"2c93772df1c5e58ab2d9ed2391c07eade76482e50b44d864b1a285aa0c9158fa"} Oct 31 03:41:23.483111 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:23.482914 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" event={"ID":"cfa0b7ad-1a4c-4803-8214-e61e9f362694","Type":"ContainerStarted","Data":"5bf06c04d4e177fec8166eaa38cc5726fcb8ce27ccf44d0d0f950455dd0f85d5"} Oct 31 03:41:23.520070 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:23.520005 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" podStartSLOduration=2.519986041 podStartE2EDuration="2.519986041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:23.519219022 +0000 UTC m=+1518.053913978" watchObservedRunningTime="2026-10-31 03:41:23.519986041 +0000 UTC m=+1518.054680997" Oct 31 03:41:23.907211 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:23.907173 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:24.256749 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:24.256637 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:24.280703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:24.277225 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:24.420609 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:24.420568 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:24.496235 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:24.496163 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:24.497351 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:24.497331 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:41:24.852179 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:24.852138 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:26.010216 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:26.010160 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:26.282849 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:26.282754 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:27.538995 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.538960 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:41:27.539463 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.539438 5490 generic.go:356] "Generic (PLEG): container finished" podID="60771f19-56cc-40a1-a27c-b09f23ac9903" containerID="78c8fca1454169c1ff41d6bc7cb33be6114c658e21bbdff35bacfc579817faa1" exitCode=0 Oct 31 03:41:27.539531 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.539507 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" event={"ID":"60771f19-56cc-40a1-a27c-b09f23ac9903","Type":"ContainerDied","Data":"78c8fca1454169c1ff41d6bc7cb33be6114c658e21bbdff35bacfc579817faa1"} Oct 31 03:41:27.539601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.539590 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" event={"ID":"60771f19-56cc-40a1-a27c-b09f23ac9903","Type":"ContainerDied","Data":"6a5f9c597dd5243adf72fdaae021da328cf87023f8b76a0370586f17617f7db1"} Oct 31 03:41:27.539641 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.539615 5490 scope.go:122] "RemoveContainer" containerID="78c8fca1454169c1ff41d6bc7cb33be6114c658e21bbdff35bacfc579817faa1" Oct 31 03:41:27.553270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.553247 5490 scope.go:122] "RemoveContainer" containerID="78c8fca1454169c1ff41d6bc7cb33be6114c658e21bbdff35bacfc579817faa1" Oct 31 03:41:27.553626 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:41:27.553602 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78c8fca1454169c1ff41d6bc7cb33be6114c658e21bbdff35bacfc579817faa1\": container with ID starting with 78c8fca1454169c1ff41d6bc7cb33be6114c658e21bbdff35bacfc579817faa1 not found: ID does not exist" containerID="78c8fca1454169c1ff41d6bc7cb33be6114c658e21bbdff35bacfc579817faa1" Oct 31 03:41:27.553710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.553644 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c8fca1454169c1ff41d6bc7cb33be6114c658e21bbdff35bacfc579817faa1"} err="failed to get container status \"78c8fca1454169c1ff41d6bc7cb33be6114c658e21bbdff35bacfc579817faa1\": rpc error: code = NotFound desc = could not find container \"78c8fca1454169c1ff41d6bc7cb33be6114c658e21bbdff35bacfc579817faa1\": container with ID starting with 78c8fca1454169c1ff41d6bc7cb33be6114c658e21bbdff35bacfc579817faa1 not found: ID does not exist" Oct 31 03:41:27.561980 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.561940 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:27.645933 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.645897 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-workload-certs\") pod \"60771f19-56cc-40a1-a27c-b09f23ac9903\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " Oct 31 03:41:27.646154 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.645988 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60771f19-56cc-40a1-a27c-b09f23ac9903-kube-api-access-qtjxw\" (UniqueName: \"kubernetes.io/projected/60771f19-56cc-40a1-a27c-b09f23ac9903-kube-api-access-qtjxw\") pod \"60771f19-56cc-40a1-a27c-b09f23ac9903\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " Oct 31 03:41:27.646154 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.646077 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-workload-socket\") pod \"60771f19-56cc-40a1-a27c-b09f23ac9903\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " Oct 31 03:41:27.646904 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.646152 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-credential-socket\") pod \"60771f19-56cc-40a1-a27c-b09f23ac9903\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " Oct 31 03:41:27.646904 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.646185 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-token\" (UniqueName: \"kubernetes.io/projected/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-token\") pod \"60771f19-56cc-40a1-a27c-b09f23ac9903\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " Oct 31 03:41:27.646904 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.646225 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-data\") pod \"60771f19-56cc-40a1-a27c-b09f23ac9903\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " Oct 31 03:41:27.646904 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.646275 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-podinfo\") pod \"60771f19-56cc-40a1-a27c-b09f23ac9903\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " Oct 31 03:41:27.646904 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.646349 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60771f19-56cc-40a1-a27c-b09f23ac9903-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60771f19-56cc-40a1-a27c-b09f23ac9903-istiod-ca-cert\") pod \"60771f19-56cc-40a1-a27c-b09f23ac9903\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " Oct 31 03:41:27.646904 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.646480 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-envoy\") pod \"60771f19-56cc-40a1-a27c-b09f23ac9903\" (UID: \"60771f19-56cc-40a1-a27c-b09f23ac9903\") " Oct 31 03:41:27.649016 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.648071 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-workload-certs" pod "60771f19-56cc-40a1-a27c-b09f23ac9903" (UID: "60771f19-56cc-40a1-a27c-b09f23ac9903"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:41:27.649867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.649842 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-workload-socket" pod "60771f19-56cc-40a1-a27c-b09f23ac9903" (UID: "60771f19-56cc-40a1-a27c-b09f23ac9903"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:41:27.650061 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.649859 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-data" pod "60771f19-56cc-40a1-a27c-b09f23ac9903" (UID: "60771f19-56cc-40a1-a27c-b09f23ac9903"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:41:27.650492 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.650470 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-credential-socket" pod "60771f19-56cc-40a1-a27c-b09f23ac9903" (UID: "60771f19-56cc-40a1-a27c-b09f23ac9903"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:41:27.651468 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.651436 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60771f19-56cc-40a1-a27c-b09f23ac9903-kube-api-access-qtjxw" pod "60771f19-56cc-40a1-a27c-b09f23ac9903" (UID: "60771f19-56cc-40a1-a27c-b09f23ac9903"). InnerVolumeSpecName "kube-api-access-qtjxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:27.652187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.652138 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60771f19-56cc-40a1-a27c-b09f23ac9903-istiod-ca-cert" pod "60771f19-56cc-40a1-a27c-b09f23ac9903" (UID: "60771f19-56cc-40a1-a27c-b09f23ac9903"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:41:27.652545 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.652518 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-envoy" pod "60771f19-56cc-40a1-a27c-b09f23ac9903" (UID: "60771f19-56cc-40a1-a27c-b09f23ac9903"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:41:27.655015 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.654983 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-podinfo" pod "60771f19-56cc-40a1-a27c-b09f23ac9903" (UID: "60771f19-56cc-40a1-a27c-b09f23ac9903"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:41:27.656754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.656731 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-token" pod "60771f19-56cc-40a1-a27c-b09f23ac9903" (UID: "60771f19-56cc-40a1-a27c-b09f23ac9903"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:41:27.749503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.749418 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:27.749503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.749461 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:27.749503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.749477 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:27.749503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.749493 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60771f19-56cc-40a1-a27c-b09f23ac9903-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:27.749782 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.749508 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:27.749782 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.749522 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:27.749782 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.749537 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtjxw\" (UniqueName: \"kubernetes.io/projected/60771f19-56cc-40a1-a27c-b09f23ac9903-kube-api-access-qtjxw\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:27.749782 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.749552 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:27.749782 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.749568 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60771f19-56cc-40a1-a27c-b09f23ac9903-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:41:27.844708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:27.830631 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:28.191348 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:28.191310 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:28.549739 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:28.549631 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r" Oct 31 03:41:28.582158 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:28.582117 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r"] Oct 31 03:41:28.587328 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:28.587299 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-w9p7r"] Oct 31 03:41:29.461982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:29.461942 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:29.580115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:29.580078 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60771f19-56cc-40a1-a27c-b09f23ac9903" path="/var/lib/kubelet/pods/60771f19-56cc-40a1-a27c-b09f23ac9903/volumes" Oct 31 03:41:29.666911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:29.666875 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:30.737188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:30.737142 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:31.773665 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:31.773620 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:32.523895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:32.523853 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:33.510261 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:33.510225 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:34.518162 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:34.518074 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:35.217099 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:35.217048 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:36.156175 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:36.156138 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:37.345221 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:37.345177 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:37.592078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:37.592036 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:38.942216 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:38.942174 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:39.873670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:39.873622 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:40.149479 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:40.149379 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:40.740595 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:40.740553 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:41.719977 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:41.719925 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:41.905157 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:41.905115 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:42.100987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:42.100899 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:42.630774 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:42.630731 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:43.511327 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:43.511284 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:44.299485 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:44.299448 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:45.123979 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:45.123932 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:45.896103 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:45.895865 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:46.623056 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:46.623016 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:47.131474 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:41:47.131436 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:41:47.518228 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:47.518190 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:48.217295 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:48.217255 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:48.977357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:48.977322 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:49.756225 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:49.756184 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:49.892914 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:49.892876 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:50.141500 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:50.141413 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:50.798263 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:50.798218 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:51.675717 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:51.675635 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:52.793290 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:52.793253 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:53.613867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:53.613172 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:54.643538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:54.643502 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:55.237529 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:55.237486 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:55.720552 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:55.720518 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:55.847869 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:55.847832 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:56.092033 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:56.089457 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:56.314090 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:56.314050 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:56.565473 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:56.565411 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:56.714589 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:56.714549 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:56.953444 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:56.953408 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6"] Oct 31 03:41:56.955084 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:56.955054 5490 cpu_manager.go:414] "containerMap: removing container" podUID="60771f19-56cc-40a1-a27c-b09f23ac9903" containerName="istio-proxy" Oct 31 03:41:56.955084 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:56.955079 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60771f19-56cc-40a1-a27c-b09f23ac9903" containerName="istio-proxy" Oct 31 03:41:56.955285 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:56.955239 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="60771f19-56cc-40a1-a27c-b09f23ac9903" containerName="istio-proxy" Oct 31 03:41:56.958945 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:56.958922 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:56.966549 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:56.966501 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-86a40d-openshift-default-dockercfg-49mvt\"" Oct 31 03:41:56.974164 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:56.974136 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6"] Oct 31 03:41:57.011520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.011478 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.011784 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.011658 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.011889 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.011819 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.011992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.011968 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.012114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.012090 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-964qm\" (UniqueName: \"kubernetes.io/projected/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-kube-api-access-964qm\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.012164 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.012151 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.012311 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.012298 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.012414 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.012324 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.012414 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.012404 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.032414 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.032342 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:57.114414 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.114377 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.114414 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.114422 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.114759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.114476 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.114759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.114522 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.114759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.114578 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.114759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.114608 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.114759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.114736 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.115048 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.114817 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-964qm\" (UniqueName: \"kubernetes.io/projected/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-kube-api-access-964qm\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.115048 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.114883 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.115048 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.114983 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.115196 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.115102 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.115309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.115278 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.115426 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.115315 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.115517 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.115501 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.117227 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.117205 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.117379 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.117360 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.139052 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.131025 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-964qm\" (UniqueName: \"kubernetes.io/projected/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-kube-api-access-964qm\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.140423 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.140340 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.263068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.262916 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:57.287459 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.287167 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:57.645249 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.645211 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:57.757009 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:41:57.756976 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0596bd0_518a_45ab_8cc1_8ba75268ef6a.slice/crio-5ec5f08e1e9e6e921d4c7e5b23d557d6f6751386c8ffb7c08b65b941dc069dd7 WatchSource:0}: Error finding container 5ec5f08e1e9e6e921d4c7e5b23d557d6f6751386c8ffb7c08b65b941dc069dd7: Status 404 returned error can't find the container with id 5ec5f08e1e9e6e921d4c7e5b23d557d6f6751386c8ffb7c08b65b941dc069dd7 Oct 31 03:41:57.757286 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.757264 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6"] Oct 31 03:41:57.760137 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.760105 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:41:57.760220 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.760172 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:41:57.760220 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:57.760201 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:41:58.273900 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:58.273843 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" event={"ID":"b0596bd0-518a-45ab-8cc1-8ba75268ef6a","Type":"ContainerStarted","Data":"f3a2648ff280ec1986d0d43efe9be84cda122733c355195a18480a5c8cd10148"} Oct 31 03:41:58.274323 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:58.273920 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" event={"ID":"b0596bd0-518a-45ab-8cc1-8ba75268ef6a","Type":"ContainerStarted","Data":"5ec5f08e1e9e6e921d4c7e5b23d557d6f6751386c8ffb7c08b65b941dc069dd7"} Oct 31 03:41:58.288795 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:58.288761 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:58.290872 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:58.290845 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:41:58.291040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:58.290894 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 31 03:41:58.311451 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:58.311406 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:58.340836 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:58.337369 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" podStartSLOduration=2.3373460489999998 podStartE2EDuration="2.337346049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:58.298162054 +0000 UTC m=+1552.832857010" watchObservedRunningTime="2026-10-31 03:41:58.337346049 +0000 UTC m=+1552.872041007" Oct 31 03:41:59.095156 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:59.095098 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:41:59.295526 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:59.295494 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:59.296486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:59.296459 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:59.304728 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:59.298961 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:41:59.411241 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:41:59.411206 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:00.067864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:00.067824 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:00.835353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:00.835312 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:00.992260 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:00.992226 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:01.153316 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:01.153283 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:01.782228 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:01.782191 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:02.262766 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:02.262734 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:02.513027 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:02.512905 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:02.951534 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:02.951484 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:03.614736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:03.614636 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:04.356228 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:04.356189 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:04.499072 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:04.499025 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:04.731072 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:04.730969 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:05.709245 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:05.704331 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:06.704053 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:06.704010 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:07.469521 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:07.469483 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:08.167697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:08.165039 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:08.436914 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:08.436823 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:08.571582 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:08.571541 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:08.908009 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:08.907960 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:09.683729 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:09.683690 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:10.397169 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:10.397111 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:10.900741 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:10.900663 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:11.110798 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:11.069284 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:11.327700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:11.326746 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:11.571599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:11.563821 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:11.886728 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:11.880467 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:12.511709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:12.500947 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:13.621744 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:13.617372 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:14.551705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:14.551242 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:15.449229 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:15.449182 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:15.711230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:15.711130 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:15.886272 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:15.886228 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:16.615235 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:16.614992 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:17.105165 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:17.105132 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:17.130216 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:42:17.130178 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:42:17.328157 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:17.328121 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:17.601181 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:17.601135 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:17.758657 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:17.758617 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:18.001178 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:18.001143 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:18.153190 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:18.153154 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:18.601763 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:18.601722 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:19.144607 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:19.144563 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:19.288028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:19.287988 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:19.431948 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:19.431904 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:19.631697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:19.624767 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:20.204491 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:20.204452 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:20.330897 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:20.330857 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:20.811556 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:20.811514 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:21.118756 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:21.117657 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:21.616769 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:21.616726 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:21.774367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:21.774330 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:22.596081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:22.596038 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:22.886243 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:22.886147 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:23.157236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:23.157204 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:23.958600 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:23.958549 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:24.106422 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:24.106380 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:24.270628 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:24.270529 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:24.912924 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:24.912880 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:25.596326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:25.596277 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:25.916704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:25.916656 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:26.069004 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:26.068967 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:26.559326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:26.559192 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:27.230131 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:27.230098 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:27.389852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:27.389805 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:27.538958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:27.538857 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:28.206468 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:28.206425 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:28.871529 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:28.871479 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:29.549535 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:29.549496 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:30.087464 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:30.087419 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:30.770550 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:30.770513 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:31.399016 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:31.398969 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:32.107859 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:32.107758 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:32.880915 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:32.880875 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:33.616703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:33.610750 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:34.311722 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:34.311668 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:35.005840 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:35.005797 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:35.731237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:35.730266 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:36.501721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:36.501656 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:37.336780 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:37.334757 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:38.478096 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:38.478056 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:39.912311 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:39.912270 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:41.236323 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:41.236282 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:42.099695 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:42.099634 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:42.914594 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:42.914555 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:43.101848 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:43.095815 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:44.498722 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:44.498666 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:45.354703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:45.348689 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:46.185534 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:46.180207 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:47.140221 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:42:47.140128 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:42:47.298845 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:47.298795 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:47.438234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:47.438194 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:48.245254 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:48.245218 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:49.479165 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:49.478648 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:50.231163 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:50.231075 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:50.986520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:50.986415 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:51.875210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:51.875167 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:52.073770 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:52.073668 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:53.463927 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:53.463878 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-586ddd746f-nv8gm" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:54.506957 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:54.506918 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:54.692479 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:54.692435 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:55.679093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:55.667747 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:42:55.756833 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:55.679303 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:55.756833 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:55.680966 5490 patch_prober.go:29] interesting pod/thanos-querier-b949d9bf6-8b6xc container/kube-rbac-proxy-web namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.133.0.10:9091/-/healthy\": context deadline exceeded" start-of-body= Oct 31 03:42:55.756833 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:55.681428 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/healthy\": context deadline exceeded" Oct 31 03:42:56.219838 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:56.219789 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:42:56.388845 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:56.388804 5490 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ttmkb container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:42:56.501264 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:56.388882 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" podUID="400b676a-a3a0-459d-9da6-8d30b2d3b497" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:57.212159 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:57.212115 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded" Oct 31 03:42:57.703808 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:57.703745 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:42:57.889306 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:57.726823 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" podUID="efd854cd-40a6-4f9f-bd19-cd5d19512dba" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.24:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:58.137871 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:58.137780 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:42:58.285306 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:58.224566 5490 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-5kbd8 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.11:9443/health\": context deadline exceeded" start-of-body= Oct 31 03:42:58.285306 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:58.224620 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" podUID="8bae2714-9c1b-44db-a6ef-b0b008cf1fa2" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.133.0.11:9443/health\": context deadline exceeded" Oct 31 03:42:58.513218 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:42:58.513159 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" podUID="55f29950-78b4-4a52-818d-dc235e4631fc" containerName="proxy" probeResult="failure" output="Get \"https://10.133.0.36:8443/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:00.384852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:00.384806 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:43:00.416642 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:00.384881 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:00.454633 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:00.418810 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:00.524861 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:00.524817 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:43:00.525034 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:00.524892 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:02.722889 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:02.722839 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" podUID="e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.101:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:03.372954 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:03.372902 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" podUID="85995108-6b7b-4e56-8260-00b5347d6e6a" containerName="postgres" probeResult="failure" output="command timed out" Oct 31 03:43:03.445698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:03.444233 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="c835066f-39c8-42b4-9f99-1055495bcc69" containerName="prometheus" probeResult="failure" output="command timed out" Oct 31 03:43:03.445698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:03.444742 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="c835066f-39c8-42b4-9f99-1055495bcc69" containerName="prometheus" probeResult="failure" output="command timed out" Oct 31 03:43:03.538400 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:03.454365 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:03.538400 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:03.458153 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" podUID="c809aea3-9195-4f90-9ffb-ca153af1a5be" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.108:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:03.538400 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:03.490266 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" podUID="fd89b915-fe3a-4a8a-aaf5-6927987b2e0d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.22:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:03.538400 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:03.528561 5490 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ttmkb container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:43:03.538400 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:03.528612 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" podUID="400b676a-a3a0-459d-9da6-8d30b2d3b497" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:03.766896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:03.766852 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" podUID="efd854cd-40a6-4f9f-bd19-cd5d19512dba" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.24:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:04.403891 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:04.403844 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:04.418593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:04.418541 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:04.562324 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:04.476824 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:04.664944 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:04.664834 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" podUID="fd89b915-fe3a-4a8a-aaf5-6927987b2e0d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.22:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:05.306873 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:05.306826 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-6r4mx" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.32:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:05.477614 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:05.477569 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 03:43:06.106743 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:43:06.106669 5490 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.19s" Oct 31 03:43:06.136390 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:06.106962 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:43:06.220381 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:06.220343 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:43:06.528382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:06.528340 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" podUID="85995108-6b7b-4e56-8260-00b5347d6e6a" containerName="postgres" probeResult="failure" output="command timed out" Oct 31 03:43:08.528906 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:08.528861 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" podUID="85995108-6b7b-4e56-8260-00b5347d6e6a" containerName="postgres" probeResult="failure" output="command timed out" Oct 31 03:43:08.615430 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:08.529060 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" Oct 31 03:43:09.600958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:09.600910 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" podUID="85995108-6b7b-4e56-8260-00b5347d6e6a" containerName="postgres" probeResult="failure" output="command timed out" Oct 31 03:43:10.548908 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:10.548864 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:43:10.549134 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:10.548946 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:14.448826 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:14.436986 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-6r4mx" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.32:8081/readyz\": dial tcp 10.133.0.32:8081: connect: connection refused" Oct 31 03:43:14.921222 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:14.921179 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:15.454082 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:15.454042 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:15.524840 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:15.524802 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:43:15.529713 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:15.524878 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:15.529713 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:15.528658 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" podUID="85995108-6b7b-4e56-8260-00b5347d6e6a" containerName="postgres" probeResult="failure" output="command timed out" Oct 31 03:43:15.723777 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:15.712336 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:15.920351 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:15.917345 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:15.994774 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:15.990711 5490 generic.go:356] "Generic (PLEG): container finished" podID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerID="7209ae60ec915af7c0e7e97853a9c90448c6f22efc9001bbe9ec2bededab41a1" exitCode=1 Oct 31 03:43:16.032399 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:15.998057 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-6r4mx" event={"ID":"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2","Type":"ContainerDied","Data":"7209ae60ec915af7c0e7e97853a9c90448c6f22efc9001bbe9ec2bededab41a1"} Oct 31 03:43:16.032399 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:16.024371 5490 scope.go:122] "RemoveContainer" containerID="7209ae60ec915af7c0e7e97853a9c90448c6f22efc9001bbe9ec2bededab41a1" Oct 31 03:43:16.032399 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:16.027294 5490 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:43:17.201180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:16.387782 5490 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ttmkb container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:43:17.201180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:16.387853 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" podUID="400b676a-a3a0-459d-9da6-8d30b2d3b497" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:17.201180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:16.452782 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:17.271491 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:17.271453 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:18.653988 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:43:18.653943 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:43:18.690558 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:18.690514 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:19.395906 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:19.395865 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:19.489609 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:19.489578 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" Oct 31 03:43:20.113603 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:20.101083 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:20.729566 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:20.700139 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:21.506702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:21.506516 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:22.347624 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:22.347580 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-6r4mx" event={"ID":"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2","Type":"ContainerStarted","Data":"d619579ca1a11c72ec2427082b17e0d482aeb731037e5af6357ac309ce0af837"} Oct 31 03:43:22.360071 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:22.347894 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-6r4mx" Oct 31 03:43:22.391992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:22.391952 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:23.169623 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:23.169579 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:24.025647 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:23.942880 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:24.092301 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:24.078484 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:24.247452 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:24.239818 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:24.418580 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:24.410451 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:24.517851 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:24.422174 5490 generic.go:356] "Generic (PLEG): container finished" podID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerID="2c93772df1c5e58ab2d9ed2391c07eade76482e50b44d864b1a285aa0c9158fa" exitCode=0 Oct 31 03:43:24.517851 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:24.422388 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" event={"ID":"cfa0b7ad-1a4c-4803-8214-e61e9f362694","Type":"ContainerDied","Data":"2c93772df1c5e58ab2d9ed2391c07eade76482e50b44d864b1a285aa0c9158fa"} Oct 31 03:43:24.517851 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:24.496908 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:43:24.590538 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:24.590455 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:24.594452 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:24.594421 5490 scope.go:122] "RemoveContainer" containerID="2c93772df1c5e58ab2d9ed2391c07eade76482e50b44d864b1a285aa0c9158fa" Oct 31 03:43:24.663081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:24.663008 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:43:24.717767 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:24.715517 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:43:24.728740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:24.721849 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:43:25.199546 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:25.199491 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:25.359193 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:25.359151 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:25.525624 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:25.525540 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:25.733367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:25.733321 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:25.934112 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:25.934068 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:26.122523 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:26.106930 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:26.443091 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:26.443052 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" event={"ID":"cfa0b7ad-1a4c-4803-8214-e61e9f362694","Type":"ContainerStarted","Data":"7cbdd276c928504a14fb06c7051ea8ff32c48ff9af01c3726baa5f4bc21ea05a"} Oct 31 03:43:27.211550 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:27.210639 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:43:27.214521 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:27.214457 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:27.215523 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:27.215487 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 03:43:27.215718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:27.215570 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 03:43:27.983571 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:27.983477 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:28.210766 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:28.210662 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 03:43:28.225301 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:28.210788 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 03:43:28.800700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:28.800421 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:29.211024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:29.210981 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 03:43:29.211444 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:29.211047 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 03:43:29.741758 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:29.725989 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:30.214223 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:30.214183 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 03:43:30.215711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:30.215645 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 03:43:30.531753 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:30.530816 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:43:30.531753 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:30.530904 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:30.854366 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:30.853352 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:31.211528 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:31.211492 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 03:43:31.211730 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:31.211550 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 03:43:32.220994 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:32.219417 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 03:43:32.220994 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:32.219497 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 03:43:32.462708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:32.458920 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:32.897328 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:32.895884 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:43:33.215698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:33.214139 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 03:43:33.215698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:33.214208 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 03:43:33.264252 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:33.263003 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:33.401704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:33.389519 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-6r4mx" Oct 31 03:43:34.220655 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:34.220613 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 03:43:34.220912 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:34.220712 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 03:43:34.222168 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:34.222004 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:35.309841 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:35.309736 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:43:35.311237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:35.310215 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:43:35.311342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:35.311293 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:43:35.577378 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:35.577282 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:35.812090 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:35.812048 5490 generic.go:356] "Generic (PLEG): container finished" podID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerID="1d426b91989be27ab5889cf4a028ee30b26e35646ac0bb25bbc43a9986204d2c" exitCode=0 Oct 31 03:43:35.814188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:35.814158 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" event={"ID":"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b","Type":"ContainerDied","Data":"1d426b91989be27ab5889cf4a028ee30b26e35646ac0bb25bbc43a9986204d2c"} Oct 31 03:43:35.815017 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:35.814997 5490 scope.go:122] "RemoveContainer" containerID="1d426b91989be27ab5889cf4a028ee30b26e35646ac0bb25bbc43a9986204d2c" Oct 31 03:43:35.818669 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:35.818594 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:43:35.819611 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:35.818773 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:43:35.819611 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:35.818886 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:43:36.463497 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:36.462926 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:36.771769 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:36.767259 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp"] Oct 31 03:43:36.771769 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:36.767977 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" podUID="113276f1-a2bd-49bd-843e-22146959715e" containerName="istio-proxy" containerID="cri-o://d5b262180510f1e0c68cc21a4af530afa514fd662ce4ea7ccd9386abef0405e9" gracePeriod=30 Oct 31 03:43:36.914240 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:36.914197 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" event={"ID":"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b","Type":"ContainerStarted","Data":"e80da2fd7fb9cb92560ec74ab6aa1a423731a1c2640319fd6b6fdb73d4762e62"} Oct 31 03:43:37.824664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:37.823724 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 31 03:43:37.824664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:37.823820 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 31 03:43:38.011388 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:38.011330 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:43:38.086646 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:38.086531 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:38.384060 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:38.383959 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:38.791703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:38.790722 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 31 03:43:38.791703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:38.790789 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 31 03:43:39.683859 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:39.681730 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:39.787502 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:39.787071 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 31 03:43:39.787502 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:39.787132 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 31 03:43:40.666281 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:40.666227 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:40.787555 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:40.787103 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 31 03:43:40.787555 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:40.787168 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 31 03:43:41.768532 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:41.768439 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:41.788159 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:41.787449 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 31 03:43:41.788159 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:41.787518 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 31 03:43:42.173110 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.173011 5490 generic.go:356] "Generic (PLEG): container finished" podID="113276f1-a2bd-49bd-843e-22146959715e" containerID="d5b262180510f1e0c68cc21a4af530afa514fd662ce4ea7ccd9386abef0405e9" exitCode=0 Oct 31 03:43:42.176830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.173435 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" event={"ID":"113276f1-a2bd-49bd-843e-22146959715e","Type":"ContainerDied","Data":"d5b262180510f1e0c68cc21a4af530afa514fd662ce4ea7ccd9386abef0405e9"} Oct 31 03:43:42.648894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.648813 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:42.656908 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.656751 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:43:42.715699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.695639 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-workload-socket\") pod \"113276f1-a2bd-49bd-843e-22146959715e\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " Oct 31 03:43:42.715699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.695809 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-istio-data\") pod \"113276f1-a2bd-49bd-843e-22146959715e\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " Oct 31 03:43:42.715699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.695940 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-credential-socket\") pod \"113276f1-a2bd-49bd-843e-22146959715e\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " Oct 31 03:43:42.715699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.696010 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-istio-envoy\") pod \"113276f1-a2bd-49bd-843e-22146959715e\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " Oct 31 03:43:42.715699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.696264 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/113276f1-a2bd-49bd-843e-22146959715e-kube-api-access-wzvg5\" (UniqueName: \"kubernetes.io/projected/113276f1-a2bd-49bd-843e-22146959715e-kube-api-access-wzvg5\") pod \"113276f1-a2bd-49bd-843e-22146959715e\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " Oct 31 03:43:42.715699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.696344 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/113276f1-a2bd-49bd-843e-22146959715e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113276f1-a2bd-49bd-843e-22146959715e-istiod-ca-cert\") pod \"113276f1-a2bd-49bd-843e-22146959715e\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " Oct 31 03:43:42.715699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.696422 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-workload-certs\") pod \"113276f1-a2bd-49bd-843e-22146959715e\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " Oct 31 03:43:42.715699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.696610 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/113276f1-a2bd-49bd-843e-22146959715e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113276f1-a2bd-49bd-843e-22146959715e-istio-podinfo\") pod \"113276f1-a2bd-49bd-843e-22146959715e\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " Oct 31 03:43:42.715699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.696690 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/113276f1-a2bd-49bd-843e-22146959715e-istio-token\" (UniqueName: \"kubernetes.io/projected/113276f1-a2bd-49bd-843e-22146959715e-istio-token\") pod \"113276f1-a2bd-49bd-843e-22146959715e\" (UID: \"113276f1-a2bd-49bd-843e-22146959715e\") " Oct 31 03:43:42.726158 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.721233 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/113276f1-a2bd-49bd-843e-22146959715e-istiod-ca-cert" pod "113276f1-a2bd-49bd-843e-22146959715e" (UID: "113276f1-a2bd-49bd-843e-22146959715e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:43:42.752329 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.728210 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-istio-data" pod "113276f1-a2bd-49bd-843e-22146959715e" (UID: "113276f1-a2bd-49bd-843e-22146959715e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:43:42.752329 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.733934 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-credential-socket" pod "113276f1-a2bd-49bd-843e-22146959715e" (UID: "113276f1-a2bd-49bd-843e-22146959715e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:43:42.752329 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.748112 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-workload-certs" pod "113276f1-a2bd-49bd-843e-22146959715e" (UID: "113276f1-a2bd-49bd-843e-22146959715e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:43:42.779607 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.761944 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-workload-socket" pod "113276f1-a2bd-49bd-843e-22146959715e" (UID: "113276f1-a2bd-49bd-843e-22146959715e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:43:42.875420 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.798012 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 31 03:43:42.875420 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.798121 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 31 03:43:42.875420 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.800497 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113276f1-a2bd-49bd-843e-22146959715e-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:42.875420 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.800529 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:42.875420 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.800545 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:42.875420 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.800561 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:42.875420 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.800574 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:42.903018 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.890100 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-istio-envoy" pod "113276f1-a2bd-49bd-843e-22146959715e" (UID: "113276f1-a2bd-49bd-843e-22146959715e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:43:42.903018 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.894283 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664"] Oct 31 03:43:42.903018 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.895703 5490 cpu_manager.go:414] "containerMap: removing container" podUID="113276f1-a2bd-49bd-843e-22146959715e" containerName="istio-proxy" Oct 31 03:43:42.903018 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.895723 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="113276f1-a2bd-49bd-843e-22146959715e" containerName="istio-proxy" Oct 31 03:43:42.903018 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.899050 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:42.903018 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.901959 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113276f1-a2bd-49bd-843e-22146959715e-istio-token" pod "113276f1-a2bd-49bd-843e-22146959715e" (UID: "113276f1-a2bd-49bd-843e-22146959715e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:43:42.903018 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.902500 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="113276f1-a2bd-49bd-843e-22146959715e" containerName="istio-proxy" Oct 31 03:43:42.905297 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.904963 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113276f1-a2bd-49bd-843e-22146959715e-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:42.905297 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.904995 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113276f1-a2bd-49bd-843e-22146959715e-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:42.912900 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.912864 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113276f1-a2bd-49bd-843e-22146959715e-kube-api-access-wzvg5" pod "113276f1-a2bd-49bd-843e-22146959715e" (UID: "113276f1-a2bd-49bd-843e-22146959715e"). InnerVolumeSpecName "kube-api-access-wzvg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:43:42.914285 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.912760 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/113276f1-a2bd-49bd-843e-22146959715e-istio-podinfo" pod "113276f1-a2bd-49bd-843e-22146959715e" (UID: "113276f1-a2bd-49bd-843e-22146959715e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:43:42.938490 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.937610 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664"] Oct 31 03:43:42.938490 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:42.937796 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.007253 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.007217 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.007513 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.007494 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.007760 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.007668 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.007880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.007865 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56k62\" (UniqueName: \"kubernetes.io/projected/a32b3b93-e0e3-4a92-9995-5d69dde3d432-kube-api-access-56k62\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.007996 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.007983 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.008130 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.008118 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.008282 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.008268 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.008513 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.008497 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.008639 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.008627 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.008916 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.008894 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzvg5\" (UniqueName: \"kubernetes.io/projected/113276f1-a2bd-49bd-843e-22146959715e-kube-api-access-wzvg5\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:43.008994 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.008921 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113276f1-a2bd-49bd-843e-22146959715e-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:43:43.110463 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.110420 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.110759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.110477 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.110759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.110563 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.110759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.110603 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56k62\" (UniqueName: \"kubernetes.io/projected/a32b3b93-e0e3-4a92-9995-5d69dde3d432-kube-api-access-56k62\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.110759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.110644 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.111011 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.110785 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.111011 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.110892 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.125148 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.111065 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.125148 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.111103 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.195574 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.195520 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:43.197278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.197231 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" event={"ID":"113276f1-a2bd-49bd-843e-22146959715e","Type":"ContainerDied","Data":"eb0db1e46d38c473025d618730c48154049cc7f8791a95028f5377dd1cd40c0f"} Oct 31 03:43:43.197565 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.197406 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp" Oct 31 03:43:43.197728 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.197539 5490 scope.go:122] "RemoveContainer" containerID="d5b262180510f1e0c68cc21a4af530afa514fd662ce4ea7ccd9386abef0405e9" Oct 31 03:43:43.282951 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.282145 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp"] Oct 31 03:43:43.294343 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.294306 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6qfqp"] Oct 31 03:43:43.380722 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.376879 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:43.437406 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.437334 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.444531 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.444483 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.445068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.445015 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.445223 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.445203 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.447968 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.447900 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56k62\" (UniqueName: \"kubernetes.io/projected/a32b3b93-e0e3-4a92-9995-5d69dde3d432-kube-api-access-56k62\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.449838 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.449782 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.449968 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.449863 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.493305 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.493269 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.502318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.502284 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.590781 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.590739 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="113276f1-a2bd-49bd-843e-22146959715e" path="/var/lib/kubelet/pods/113276f1-a2bd-49bd-843e-22146959715e/volumes" Oct 31 03:43:43.725203 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.725122 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:43.772405 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.772369 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:43.791866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.791798 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:43:43.792262 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.792233 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:43:43.793553 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:43.793528 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:43:44.059800 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:44.045066 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:44.561659 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:44.561626 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664"] Oct 31 03:43:44.576583 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:43:44.576543 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32b3b93_e0e3_4a92_9995_5d69dde3d432.slice/crio-893e31b4258457d439e1503fa705ee5fa5e26281920eecf51375f455e9dabbea WatchSource:0}: Error finding container 893e31b4258457d439e1503fa705ee5fa5e26281920eecf51375f455e9dabbea: Status 404 returned error can't find the container with id 893e31b4258457d439e1503fa705ee5fa5e26281920eecf51375f455e9dabbea Oct 31 03:43:44.579557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:44.579523 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:43:44.579692 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:44.579599 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:43:44.579692 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:44.579654 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:43:45.342895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:45.342852 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:43:45.343359 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:45.342936 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:43:45.422011 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:45.367594 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:45.453473 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:45.453430 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" event={"ID":"a32b3b93-e0e3-4a92-9995-5d69dde3d432","Type":"ContainerStarted","Data":"368757071fa804f4f8be23437f5661754eb6d16009155bae7e25b457aae1dc90"} Oct 31 03:43:45.453758 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:45.453504 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" event={"ID":"a32b3b93-e0e3-4a92-9995-5d69dde3d432","Type":"ContainerStarted","Data":"893e31b4258457d439e1503fa705ee5fa5e26281920eecf51375f455e9dabbea"} Oct 31 03:43:45.504830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:45.502760 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" podStartSLOduration=3.502738572 podStartE2EDuration="3.502738572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:45.493151457 +0000 UTC m=+1660.027846472" watchObservedRunningTime="2026-10-31 03:43:45.502738572 +0000 UTC m=+1660.037433527" Oct 31 03:43:45.725524 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:45.725481 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:45.757880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:45.742810 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:43:45.757880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:45.742880 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" podUID="a32b3b93-e0e3-4a92-9995-5d69dde3d432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:43:46.210158 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:46.209815 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:46.730710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:46.729829 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:43:46.730710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:46.729921 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" podUID="a32b3b93-e0e3-4a92-9995-5d69dde3d432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:43:47.384802 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:43:47.384755 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:43:47.582701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:47.573473 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:47.726594 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:47.726143 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:43:47.726594 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:47.726216 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" podUID="a32b3b93-e0e3-4a92-9995-5d69dde3d432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:43:48.725559 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:48.725522 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:43:48.726165 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:48.725587 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" podUID="a32b3b93-e0e3-4a92-9995-5d69dde3d432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:43:48.892042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:48.891969 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:49.553713 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:49.553168 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" podUID="85995108-6b7b-4e56-8260-00b5347d6e6a" containerName="postgres" probeResult="failure" output="command timed out" Oct 31 03:43:49.731719 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:49.731051 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:43:49.731719 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:49.731128 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" podUID="a32b3b93-e0e3-4a92-9995-5d69dde3d432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:43:49.869783 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:49.861413 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:50.729739 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:50.726050 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 31 03:43:50.729739 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:50.726115 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" podUID="a32b3b93-e0e3-4a92-9995-5d69dde3d432" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 31 03:43:50.866767 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:50.866728 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:51.733117 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:51.733082 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:51.733583 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:51.733561 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:51.734288 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:51.734266 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:43:51.916692 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:51.916567 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:52.607786 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:52.603703 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:52.770083 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:52.770045 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:53.448918 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:53.448878 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:54.301412 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:54.301366 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:55.216409 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:55.216330 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:55.936162 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:55.936111 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:56.587294 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:56.587264 5490 generic.go:356] "Generic (PLEG): container finished" podID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerID="f3a2648ff280ec1986d0d43efe9be84cda122733c355195a18480a5c8cd10148" exitCode=0 Oct 31 03:43:56.587770 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:56.587338 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" event={"ID":"b0596bd0-518a-45ab-8cc1-8ba75268ef6a","Type":"ContainerDied","Data":"f3a2648ff280ec1986d0d43efe9be84cda122733c355195a18480a5c8cd10148"} Oct 31 03:43:56.587963 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:56.587944 5490 scope.go:122] "RemoveContainer" containerID="f3a2648ff280ec1986d0d43efe9be84cda122733c355195a18480a5c8cd10148" Oct 31 03:43:56.590593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:56.590562 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:43:56.590715 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:56.590640 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:43:56.590715 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:56.590686 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:43:56.665267 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:56.665187 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:57.398333 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:57.398301 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:57.600202 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:57.600160 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" event={"ID":"b0596bd0-518a-45ab-8cc1-8ba75268ef6a","Type":"ContainerStarted","Data":"366e1f7adbc32af7df7d579b46e5dbc7616a15291c03d27b78295a678b5b5977"} Oct 31 03:43:58.178658 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:58.175481 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:58.288269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:58.288240 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:43:58.289058 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:58.289025 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:43:58.289213 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:58.289082 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 31 03:43:58.944325 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:58.944290 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:43:59.288309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:59.288219 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:43:59.288309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:59.288285 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 31 03:43:59.296545 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:59.296499 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:43:59.892791 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:43:59.892757 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:00.288148 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:00.288106 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:44:00.288563 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:00.288174 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 31 03:44:00.613595 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:00.613505 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:01.288603 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:01.288566 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:44:01.289065 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:01.288624 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 31 03:44:01.445799 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:01.445759 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:02.288637 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:02.288596 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:44:02.289080 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:02.288664 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 31 03:44:02.375064 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:02.375001 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:02.807731 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:02.805825 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:03.288734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:03.288694 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 31 03:44:03.289128 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:03.288765 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 31 03:44:03.410853 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:03.410813 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:04.252148 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:04.252108 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:04.312023 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:04.311987 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:44:04.314305 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:04.314275 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:44:05.114364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:05.114322 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:05.589936 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:05.589904 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s"] Oct 31 03:44:05.590428 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:05.590104 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" podUID="7658fa72-7a26-4904-b702-6006f741af65" containerName="istio-proxy" containerID="cri-o://a08686043fc4d038cd9d6b8bc4697bb97c7a49f47f0a4cb09fcb72e101d0ffe9" gracePeriod=30 Oct 31 03:44:05.937080 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:05.937038 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:06.448937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:06.448903 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22"] Oct 31 03:44:06.449204 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:06.449140 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" podUID="0a305cab-5370-4be7-a94b-e2c8bb849726" containerName="istio-proxy" containerID="cri-o://81368f81741a3f5af04196f3bd7fd1586f9c6968340c7a4cafbf583f51637a62" gracePeriod=30 Oct 31 03:44:06.748402 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:06.748277 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:07.583007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:07.582962 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:08.311273 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:08.311229 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:08.993643 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:08.993601 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:09.155942 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:09.154351 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:09.840712 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:09.836806 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:10.586214 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:10.586176 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:10.800247 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:10.800206 5490 generic.go:356] "Generic (PLEG): container finished" podID="7658fa72-7a26-4904-b702-6006f741af65" containerID="a08686043fc4d038cd9d6b8bc4697bb97c7a49f47f0a4cb09fcb72e101d0ffe9" exitCode=0 Oct 31 03:44:10.801015 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:10.800278 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" event={"ID":"7658fa72-7a26-4904-b702-6006f741af65","Type":"ContainerDied","Data":"a08686043fc4d038cd9d6b8bc4697bb97c7a49f47f0a4cb09fcb72e101d0ffe9"} Oct 31 03:44:10.967603 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:10.967575 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:44:11.071506 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.071467 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v"] Oct 31 03:44:11.072617 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.072588 5490 cpu_manager.go:414] "containerMap: removing container" podUID="7658fa72-7a26-4904-b702-6006f741af65" containerName="istio-proxy" Oct 31 03:44:11.072617 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.072615 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7658fa72-7a26-4904-b702-6006f741af65" containerName="istio-proxy" Oct 31 03:44:11.072945 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.072925 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="7658fa72-7a26-4904-b702-6006f741af65" containerName="istio-proxy" Oct 31 03:44:11.077731 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.077703 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.103625 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.103585 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7658fa72-7a26-4904-b702-6006f741af65-istio-token\" (UniqueName: \"kubernetes.io/projected/7658fa72-7a26-4904-b702-6006f741af65-istio-token\") pod \"7658fa72-7a26-4904-b702-6006f741af65\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " Oct 31 03:44:11.103833 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.103654 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-istio-envoy\") pod \"7658fa72-7a26-4904-b702-6006f741af65\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " Oct 31 03:44:11.103833 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.103724 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-workload-certs\") pod \"7658fa72-7a26-4904-b702-6006f741af65\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " Oct 31 03:44:11.103833 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.103753 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7658fa72-7a26-4904-b702-6006f741af65-kube-api-access-br78q\" (UniqueName: \"kubernetes.io/projected/7658fa72-7a26-4904-b702-6006f741af65-kube-api-access-br78q\") pod \"7658fa72-7a26-4904-b702-6006f741af65\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " Oct 31 03:44:11.104069 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.104050 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-workload-socket\") pod \"7658fa72-7a26-4904-b702-6006f741af65\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " Oct 31 03:44:11.104133 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.104099 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7658fa72-7a26-4904-b702-6006f741af65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7658fa72-7a26-4904-b702-6006f741af65-istiod-ca-cert\") pod \"7658fa72-7a26-4904-b702-6006f741af65\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " Oct 31 03:44:11.104196 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.104143 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-istio-data\") pod \"7658fa72-7a26-4904-b702-6006f741af65\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " Oct 31 03:44:11.104196 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.104183 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7658fa72-7a26-4904-b702-6006f741af65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7658fa72-7a26-4904-b702-6006f741af65-istio-podinfo\") pod \"7658fa72-7a26-4904-b702-6006f741af65\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " Oct 31 03:44:11.104321 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.104306 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-credential-socket\") pod \"7658fa72-7a26-4904-b702-6006f741af65\" (UID: \"7658fa72-7a26-4904-b702-6006f741af65\") " Oct 31 03:44:11.106772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.104797 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-istio-data" pod "7658fa72-7a26-4904-b702-6006f741af65" (UID: "7658fa72-7a26-4904-b702-6006f741af65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:11.106772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.104818 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-workload-socket" pod "7658fa72-7a26-4904-b702-6006f741af65" (UID: "7658fa72-7a26-4904-b702-6006f741af65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:11.106772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.105107 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7658fa72-7a26-4904-b702-6006f741af65-istiod-ca-cert" pod "7658fa72-7a26-4904-b702-6006f741af65" (UID: "7658fa72-7a26-4904-b702-6006f741af65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:44:11.106772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.105359 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:11.106772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.105370 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-workload-certs" pod "7658fa72-7a26-4904-b702-6006f741af65" (UID: "7658fa72-7a26-4904-b702-6006f741af65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:11.106772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.105384 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7658fa72-7a26-4904-b702-6006f741af65-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:11.106772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.105402 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:11.108854 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.108828 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-credential-socket" pod "7658fa72-7a26-4904-b702-6006f741af65" (UID: "7658fa72-7a26-4904-b702-6006f741af65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:11.111956 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.111891 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v"] Oct 31 03:44:11.113929 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.113904 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7658fa72-7a26-4904-b702-6006f741af65-istio-podinfo" pod "7658fa72-7a26-4904-b702-6006f741af65" (UID: "7658fa72-7a26-4904-b702-6006f741af65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:44:11.117114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.117089 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-istio-envoy" pod "7658fa72-7a26-4904-b702-6006f741af65" (UID: "7658fa72-7a26-4904-b702-6006f741af65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:11.117775 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.117668 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7658fa72-7a26-4904-b702-6006f741af65-kube-api-access-br78q" pod "7658fa72-7a26-4904-b702-6006f741af65" (UID: "7658fa72-7a26-4904-b702-6006f741af65"). InnerVolumeSpecName "kube-api-access-br78q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:11.119855 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.119794 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7658fa72-7a26-4904-b702-6006f741af65-istio-token" pod "7658fa72-7a26-4904-b702-6006f741af65" (UID: "7658fa72-7a26-4904-b702-6006f741af65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:11.206484 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.206372 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.206686 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.206515 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.206686 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.206576 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.206686 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.206635 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.206686 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.206730 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca122c09-e327-4371-9879-0870b37db17f-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.207478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.206867 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dslk5\" (UniqueName: \"kubernetes.io/projected/ca122c09-e327-4371-9879-0870b37db17f-kube-api-access-dslk5\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.207478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.206956 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca122c09-e327-4371-9879-0870b37db17f-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.207478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.207112 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca122c09-e327-4371-9879-0870b37db17f-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.207478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.207210 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.207478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.207327 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7658fa72-7a26-4904-b702-6006f741af65-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:11.207478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.207419 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:11.207478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.207444 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:11.207478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.207459 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br78q\" (UniqueName: \"kubernetes.io/projected/7658fa72-7a26-4904-b702-6006f741af65-kube-api-access-br78q\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:11.207478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.207480 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7658fa72-7a26-4904-b702-6006f741af65-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:11.208607 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.207494 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7658fa72-7a26-4904-b702-6006f741af65-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:11.309569 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.309530 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dslk5\" (UniqueName: \"kubernetes.io/projected/ca122c09-e327-4371-9879-0870b37db17f-kube-api-access-dslk5\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.309852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.309663 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca122c09-e327-4371-9879-0870b37db17f-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.309852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.309785 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca122c09-e327-4371-9879-0870b37db17f-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.309997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.309884 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.309997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.309979 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.310104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.310042 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.310104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.310088 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.310215 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.310113 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.310215 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.310188 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca122c09-e327-4371-9879-0870b37db17f-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.310946 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.310909 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.310946 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.310940 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.311132 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.310994 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.311132 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.310992 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca122c09-e327-4371-9879-0870b37db17f-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.311132 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.311082 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.313632 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.313606 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.313854 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.313835 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca122c09-e327-4371-9879-0870b37db17f-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.319932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.319907 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca122c09-e327-4371-9879-0870b37db17f-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.320096 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.320076 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dslk5\" (UniqueName: \"kubernetes.io/projected/ca122c09-e327-4371-9879-0870b37db17f-kube-api-access-dslk5\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.413900 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.413854 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:11.499622 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.499450 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:11.762976 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.762944 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v"] Oct 31 03:44:11.764791 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:44:11.764753 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca122c09_e327_4371_9879_0870b37db17f.slice/crio-d8abd9a9d6e9259a3d1ab52982ebafac2b71267ed995f3c76315e17d4fe6e09c WatchSource:0}: Error finding container d8abd9a9d6e9259a3d1ab52982ebafac2b71267ed995f3c76315e17d4fe6e09c: Status 404 returned error can't find the container with id d8abd9a9d6e9259a3d1ab52982ebafac2b71267ed995f3c76315e17d4fe6e09c Oct 31 03:44:11.769187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.769143 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:44:11.769297 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.769248 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:44:11.769369 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.769297 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:44:11.822434 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.822393 5490 generic.go:356] "Generic (PLEG): container finished" podID="0a305cab-5370-4be7-a94b-e2c8bb849726" containerID="81368f81741a3f5af04196f3bd7fd1586f9c6968340c7a4cafbf583f51637a62" exitCode=0 Oct 31 03:44:11.822627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.822499 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" event={"ID":"0a305cab-5370-4be7-a94b-e2c8bb849726","Type":"ContainerDied","Data":"81368f81741a3f5af04196f3bd7fd1586f9c6968340c7a4cafbf583f51637a62"} Oct 31 03:44:11.824504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.824473 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" event={"ID":"ca122c09-e327-4371-9879-0870b37db17f","Type":"ContainerStarted","Data":"d8abd9a9d6e9259a3d1ab52982ebafac2b71267ed995f3c76315e17d4fe6e09c"} Oct 31 03:44:11.826419 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.826394 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" Oct 31 03:44:11.826519 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.826393 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s" event={"ID":"7658fa72-7a26-4904-b702-6006f741af65","Type":"ContainerDied","Data":"a4346efeefce4787f837ae6ab78acd2b722abeb04f65651b27fba06761d81a56"} Oct 31 03:44:11.826575 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.826534 5490 scope.go:122] "RemoveContainer" containerID="a08686043fc4d038cd9d6b8bc4697bb97c7a49f47f0a4cb09fcb72e101d0ffe9" Oct 31 03:44:11.852305 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.852267 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s"] Oct 31 03:44:11.857149 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.857119 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-8h46s"] Oct 31 03:44:11.879825 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.879799 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:44:11.929709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.926772 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-token\" (UniqueName: \"kubernetes.io/projected/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-token\") pod \"0a305cab-5370-4be7-a94b-e2c8bb849726\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " Oct 31 03:44:11.929709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.926941 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-workload-certs\") pod \"0a305cab-5370-4be7-a94b-e2c8bb849726\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " Oct 31 03:44:11.929709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.927055 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-workload-socket\") pod \"0a305cab-5370-4be7-a94b-e2c8bb849726\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " Oct 31 03:44:11.929709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.927101 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-envoy\") pod \"0a305cab-5370-4be7-a94b-e2c8bb849726\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " Oct 31 03:44:11.929709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.927144 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-credential-socket\") pod \"0a305cab-5370-4be7-a94b-e2c8bb849726\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " Oct 31 03:44:11.929709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.927241 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a305cab-5370-4be7-a94b-e2c8bb849726-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a305cab-5370-4be7-a94b-e2c8bb849726-istiod-ca-cert\") pod \"0a305cab-5370-4be7-a94b-e2c8bb849726\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " Oct 31 03:44:11.929709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.927372 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-podinfo\") pod \"0a305cab-5370-4be7-a94b-e2c8bb849726\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " Oct 31 03:44:11.929709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.927428 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a305cab-5370-4be7-a94b-e2c8bb849726-kube-api-access-pgrtc\" (UniqueName: \"kubernetes.io/projected/0a305cab-5370-4be7-a94b-e2c8bb849726-kube-api-access-pgrtc\") pod \"0a305cab-5370-4be7-a94b-e2c8bb849726\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " Oct 31 03:44:11.929709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.927557 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-data\") pod \"0a305cab-5370-4be7-a94b-e2c8bb849726\" (UID: \"0a305cab-5370-4be7-a94b-e2c8bb849726\") " Oct 31 03:44:11.932018 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.931810 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-data" pod "0a305cab-5370-4be7-a94b-e2c8bb849726" (UID: "0a305cab-5370-4be7-a94b-e2c8bb849726"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:11.932360 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.932173 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-workload-certs" pod "0a305cab-5370-4be7-a94b-e2c8bb849726" (UID: "0a305cab-5370-4be7-a94b-e2c8bb849726"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:11.934407 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.934324 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-workload-socket" pod "0a305cab-5370-4be7-a94b-e2c8bb849726" (UID: "0a305cab-5370-4be7-a94b-e2c8bb849726"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:11.944920 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.944820 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-credential-socket" pod "0a305cab-5370-4be7-a94b-e2c8bb849726" (UID: "0a305cab-5370-4be7-a94b-e2c8bb849726"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:11.948789 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.945469 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a305cab-5370-4be7-a94b-e2c8bb849726-istiod-ca-cert" pod "0a305cab-5370-4be7-a94b-e2c8bb849726" (UID: "0a305cab-5370-4be7-a94b-e2c8bb849726"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:44:11.951491 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.951430 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-envoy" pod "0a305cab-5370-4be7-a94b-e2c8bb849726" (UID: "0a305cab-5370-4be7-a94b-e2c8bb849726"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:11.956018 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.955488 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-token" pod "0a305cab-5370-4be7-a94b-e2c8bb849726" (UID: "0a305cab-5370-4be7-a94b-e2c8bb849726"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:11.966234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.964426 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn"] Oct 31 03:44:11.966234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.965935 5490 cpu_manager.go:414] "containerMap: removing container" podUID="0a305cab-5370-4be7-a94b-e2c8bb849726" containerName="istio-proxy" Oct 31 03:44:11.966234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.965957 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a305cab-5370-4be7-a94b-e2c8bb849726" containerName="istio-proxy" Oct 31 03:44:11.966234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.966140 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a305cab-5370-4be7-a94b-e2c8bb849726" containerName="istio-proxy" Oct 31 03:44:11.969304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.969273 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a305cab-5370-4be7-a94b-e2c8bb849726-kube-api-access-pgrtc" pod "0a305cab-5370-4be7-a94b-e2c8bb849726" (UID: "0a305cab-5370-4be7-a94b-e2c8bb849726"). InnerVolumeSpecName "kube-api-access-pgrtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:11.973009 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.972988 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:11.974493 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.974365 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-podinfo" pod "0a305cab-5370-4be7-a94b-e2c8bb849726" (UID: "0a305cab-5370-4be7-a94b-e2c8bb849726"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:44:11.977332 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.976584 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn"] Oct 31 03:44:11.984315 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:11.983346 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-46ebfe-openshift-default-dockercfg-zlk2v\"" Oct 31 03:44:12.031504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.030489 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-workload-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.031504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.030588 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr7lb\" (UniqueName: \"kubernetes.io/projected/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-kube-api-access-qr7lb\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.031504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.030636 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istiod-ca-cert\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.031504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.030667 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-podinfo\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.031504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.030742 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-token\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.031504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.030839 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-workload-certs\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.031504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.030905 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-envoy\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.031504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.031012 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-data\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.031504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.031236 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-credential-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.031504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.031329 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:12.031504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.031339 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:12.031504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.031349 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:12.031504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.031359 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:12.031504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.031369 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a305cab-5370-4be7-a94b-e2c8bb849726-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:12.031504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.031378 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:12.031504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.031387 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgrtc\" (UniqueName: \"kubernetes.io/projected/0a305cab-5370-4be7-a94b-e2c8bb849726-kube-api-access-pgrtc\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:12.031504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.031397 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:12.031504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.031407 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a305cab-5370-4be7-a94b-e2c8bb849726-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:12.132802 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.132753 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-credential-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.133011 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.132949 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-workload-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.133011 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.132984 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr7lb\" (UniqueName: \"kubernetes.io/projected/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-kube-api-access-qr7lb\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.133145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.133029 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istiod-ca-cert\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.133145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.133058 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-podinfo\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.133145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.133089 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-token\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.133309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.133183 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-workload-certs\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.133309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.133207 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-credential-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.133309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.133248 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-envoy\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.133309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.133304 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-data\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.133796 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.133774 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-data\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.133871 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.133851 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istiod-ca-cert\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.134112 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.134093 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-workload-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.135026 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.135006 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-workload-certs\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.137004 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.136982 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-podinfo\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.137748 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.137723 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-envoy\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.142841 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.142806 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-token\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.143033 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.143013 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr7lb\" (UniqueName: \"kubernetes.io/projected/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-kube-api-access-qr7lb\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.304407 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.304324 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:12.454521 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.454473 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:12.617371 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.617336 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn"] Oct 31 03:44:12.617746 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:44:12.617723 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb295e52_5a48_4ec9_a09c_83f8d4b9f62c.slice/crio-2f6b3d00b62c4b05496eb21f1cbf6581f9ccce68c0531c5278f551733f67d45d WatchSource:0}: Error finding container 2f6b3d00b62c4b05496eb21f1cbf6581f9ccce68c0531c5278f551733f67d45d: Status 404 returned error can't find the container with id 2f6b3d00b62c4b05496eb21f1cbf6581f9ccce68c0531c5278f551733f67d45d Oct 31 03:44:12.620947 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.620913 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:44:12.621024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.620987 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:44:12.621024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.621019 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:44:12.965699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.963790 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" event={"ID":"0a305cab-5370-4be7-a94b-e2c8bb849726","Type":"ContainerDied","Data":"6db6698983cca634bfc1fa1eccb9b6f6b7c9fa60a092203386d35a82302abed8"} Oct 31 03:44:12.965699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.963879 5490 scope.go:122] "RemoveContainer" containerID="81368f81741a3f5af04196f3bd7fd1586f9c6968340c7a4cafbf583f51637a62" Oct 31 03:44:12.965699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.964023 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22" Oct 31 03:44:12.987102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:12.987028 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" event={"ID":"ca122c09-e327-4371-9879-0870b37db17f","Type":"ContainerStarted","Data":"9fadd77f20e2703cec2929d2902eeba80251d191aebdab611d75537924d9e080"} Oct 31 03:44:13.002559 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:13.002505 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" event={"ID":"db295e52-5a48-4ec9-a09c-83f8d4b9f62c","Type":"ContainerStarted","Data":"b386d1b71d98893e74885db42759861c23152f11c16b036cbda9a946f2918c88"} Oct 31 03:44:13.002559 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:13.002563 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" event={"ID":"db295e52-5a48-4ec9-a09c-83f8d4b9f62c","Type":"ContainerStarted","Data":"2f6b3d00b62c4b05496eb21f1cbf6581f9ccce68c0531c5278f551733f67d45d"} Oct 31 03:44:13.017391 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:13.017343 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" podStartSLOduration=31.017326799 podStartE2EDuration="31.017326799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:13.012787811 +0000 UTC m=+1687.547482790" watchObservedRunningTime="2026-10-31 03:44:13.017326799 +0000 UTC m=+1687.552021754" Oct 31 03:44:13.076743 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:13.069801 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" podStartSLOduration=25.069778056 podStartE2EDuration="25.069778056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:13.055999961 +0000 UTC m=+1687.590694917" watchObservedRunningTime="2026-10-31 03:44:13.069778056 +0000 UTC m=+1687.604473012" Oct 31 03:44:13.088283 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:13.088254 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22"] Oct 31 03:44:13.097333 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:13.097288 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-q5w22"] Oct 31 03:44:13.305521 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:13.305438 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:13.308011 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:13.307960 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:44:13.308153 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:13.308041 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" podUID="db295e52-5a48-4ec9-a09c-83f8d4b9f62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:44:13.424813 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:13.424760 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:13.453426 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:13.453367 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:44:13.453769 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:13.453722 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" podUID="ca122c09-e327-4371-9879-0870b37db17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:44:13.466624 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:13.466590 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:13.600285 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:13.600184 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a305cab-5370-4be7-a94b-e2c8bb849726" path="/var/lib/kubelet/pods/0a305cab-5370-4be7-a94b-e2c8bb849726/volumes" Oct 31 03:44:13.601159 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:13.601131 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7658fa72-7a26-4904-b702-6006f741af65" path="/var/lib/kubelet/pods/7658fa72-7a26-4904-b702-6006f741af65/volumes" Oct 31 03:44:14.305554 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:14.305520 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:44:14.306039 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:14.306009 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" podUID="db295e52-5a48-4ec9-a09c-83f8d4b9f62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:44:14.414509 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:14.414465 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:44:14.414723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:14.414555 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" podUID="ca122c09-e327-4371-9879-0870b37db17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:44:14.580788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:14.577909 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx"] Oct 31 03:44:14.584700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:14.583644 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" podUID="11c5b443-e562-448f-a540-2273f5ae86ca" containerName="istio-proxy" containerID="cri-o://d37131b9f898f21b1425d43addcf6b69f1590f3344ea08bb6a598180a05fdf32" gracePeriod=30 Oct 31 03:44:14.701827 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:14.701789 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:15.172923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:15.172887 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:15.306701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:15.306650 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:44:15.307161 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:15.307133 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" podUID="db295e52-5a48-4ec9-a09c-83f8d4b9f62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:44:15.386434 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:15.386405 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:15.414203 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:15.414163 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:44:15.414381 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:15.414231 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" podUID="ca122c09-e327-4371-9879-0870b37db17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:44:16.142708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:16.137191 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:16.305207 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:16.305166 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:44:16.305388 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:16.305226 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" podUID="db295e52-5a48-4ec9-a09c-83f8d4b9f62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:44:16.414405 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:16.414368 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:44:16.414859 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:16.414427 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" podUID="ca122c09-e327-4371-9879-0870b37db17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:44:16.600712 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:16.598505 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:16.818756 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:16.796854 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:17.311700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:17.307257 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:44:17.311700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:17.307352 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" podUID="db295e52-5a48-4ec9-a09c-83f8d4b9f62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:44:17.410473 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:17.410383 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:17.414339 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:44:17.413930 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:44:17.414339 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:17.414258 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:44:17.414339 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:17.414327 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" podUID="ca122c09-e327-4371-9879-0870b37db17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:44:17.622760 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:17.608556 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:18.306718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:18.306629 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:44:18.306933 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:18.306743 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" podUID="db295e52-5a48-4ec9-a09c-83f8d4b9f62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:44:18.423706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:18.421530 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:44:18.423706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:18.421615 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" podUID="ca122c09-e327-4371-9879-0870b37db17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:44:18.697697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:18.690202 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:19.314698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:19.314130 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:44:19.314698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:19.314195 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" podUID="db295e52-5a48-4ec9-a09c-83f8d4b9f62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:44:19.418700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:19.416119 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 03:44:19.418700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:19.416181 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" podUID="ca122c09-e327-4371-9879-0870b37db17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 03:44:20.271803 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:20.271762 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:20.360321 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:20.333610 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 03:44:20.382898 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:20.360606 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" podUID="db295e52-5a48-4ec9-a09c-83f8d4b9f62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 03:44:20.478702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:20.477808 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:44:20.503153 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:20.502782 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-9wbqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 31 03:44:20.503153 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:20.502847 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" podUID="11c5b443-e562-448f-a540-2273f5ae86ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 31 03:44:20.578370 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:20.577830 5490 generic.go:356] "Generic (PLEG): container finished" podID="11c5b443-e562-448f-a540-2273f5ae86ca" containerID="d37131b9f898f21b1425d43addcf6b69f1590f3344ea08bb6a598180a05fdf32" exitCode=0 Oct 31 03:44:20.578370 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:20.578208 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" event={"ID":"11c5b443-e562-448f-a540-2273f5ae86ca","Type":"ContainerDied","Data":"d37131b9f898f21b1425d43addcf6b69f1590f3344ea08bb6a598180a05fdf32"} Oct 31 03:44:20.770820 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:20.760004 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:44:20.934027 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:20.933109 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:44:20.936348 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:20.936309 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:44:20.998870 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:20.998638 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:44:21.038205 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.038160 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:21.042349 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.040317 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:21.042349 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.042045 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:44:21.046860 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.044344 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s"] Oct 31 03:44:21.046860 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.045545 5490 cpu_manager.go:414] "containerMap: removing container" podUID="11c5b443-e562-448f-a540-2273f5ae86ca" containerName="istio-proxy" Oct 31 03:44:21.046860 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.045578 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c5b443-e562-448f-a540-2273f5ae86ca" containerName="istio-proxy" Oct 31 03:44:21.046860 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.045826 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c5b443-e562-448f-a540-2273f5ae86ca" containerName="istio-proxy" Oct 31 03:44:21.071156 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.053287 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.104019 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.103979 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s"] Oct 31 03:44:21.169456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.160184 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11c5b443-e562-448f-a540-2273f5ae86ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c5b443-e562-448f-a540-2273f5ae86ca-istio-podinfo\") pod \"11c5b443-e562-448f-a540-2273f5ae86ca\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " Oct 31 03:44:21.169456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.160317 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c5b443-e562-448f-a540-2273f5ae86ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c5b443-e562-448f-a540-2273f5ae86ca-istiod-ca-cert\") pod \"11c5b443-e562-448f-a540-2273f5ae86ca\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " Oct 31 03:44:21.169456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.160394 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-istio-envoy\") pod \"11c5b443-e562-448f-a540-2273f5ae86ca\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " Oct 31 03:44:21.169456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.160488 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c5b443-e562-448f-a540-2273f5ae86ca-kube-api-access-cfcmm\" (UniqueName: \"kubernetes.io/projected/11c5b443-e562-448f-a540-2273f5ae86ca-kube-api-access-cfcmm\") pod \"11c5b443-e562-448f-a540-2273f5ae86ca\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " Oct 31 03:44:21.169456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.160521 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c5b443-e562-448f-a540-2273f5ae86ca-istio-token\" (UniqueName: \"kubernetes.io/projected/11c5b443-e562-448f-a540-2273f5ae86ca-istio-token\") pod \"11c5b443-e562-448f-a540-2273f5ae86ca\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " Oct 31 03:44:21.169456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.160552 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-credential-socket\") pod \"11c5b443-e562-448f-a540-2273f5ae86ca\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " Oct 31 03:44:21.169456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.160638 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-workload-socket\") pod \"11c5b443-e562-448f-a540-2273f5ae86ca\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " Oct 31 03:44:21.169456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.160728 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-workload-certs\") pod \"11c5b443-e562-448f-a540-2273f5ae86ca\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " Oct 31 03:44:21.169456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.160784 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-istio-data\") pod \"11c5b443-e562-448f-a540-2273f5ae86ca\" (UID: \"11c5b443-e562-448f-a540-2273f5ae86ca\") " Oct 31 03:44:21.169456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.161064 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.169456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.161095 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.169456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.161244 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.169456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.161258 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.169456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.161339 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz6wf\" (UniqueName: \"kubernetes.io/projected/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-kube-api-access-bz6wf\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.169456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.161423 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.169456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.161467 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.169456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.161543 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.169456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.161635 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.169456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.164289 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c5b443-e562-448f-a540-2273f5ae86ca-istiod-ca-cert" pod "11c5b443-e562-448f-a540-2273f5ae86ca" (UID: "11c5b443-e562-448f-a540-2273f5ae86ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:44:21.169456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.165312 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-workload-socket" pod "11c5b443-e562-448f-a540-2273f5ae86ca" (UID: "11c5b443-e562-448f-a540-2273f5ae86ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:21.183875 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.174017 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c5b443-e562-448f-a540-2273f5ae86ca-kube-api-access-cfcmm" pod "11c5b443-e562-448f-a540-2273f5ae86ca" (UID: "11c5b443-e562-448f-a540-2273f5ae86ca"). InnerVolumeSpecName "kube-api-access-cfcmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:21.183875 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.177599 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11c5b443-e562-448f-a540-2273f5ae86ca-istio-podinfo" pod "11c5b443-e562-448f-a540-2273f5ae86ca" (UID: "11c5b443-e562-448f-a540-2273f5ae86ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:44:21.201750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.200612 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-workload-certs" pod "11c5b443-e562-448f-a540-2273f5ae86ca" (UID: "11c5b443-e562-448f-a540-2273f5ae86ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:21.204645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.204591 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-credential-socket" pod "11c5b443-e562-448f-a540-2273f5ae86ca" (UID: "11c5b443-e562-448f-a540-2273f5ae86ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:21.205266 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.205241 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-istio-data" pod "11c5b443-e562-448f-a540-2273f5ae86ca" (UID: "11c5b443-e562-448f-a540-2273f5ae86ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:21.208038 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.207997 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c5b443-e562-448f-a540-2273f5ae86ca-istio-token" pod "11c5b443-e562-448f-a540-2273f5ae86ca" (UID: "11c5b443-e562-448f-a540-2273f5ae86ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:21.214253 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.214218 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-istio-envoy" pod "11c5b443-e562-448f-a540-2273f5ae86ca" (UID: "11c5b443-e562-448f-a540-2273f5ae86ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:21.265734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.263562 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.265734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.264038 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.265734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.264185 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.265734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.264874 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.265734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.264921 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.265734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.265033 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz6wf\" (UniqueName: \"kubernetes.io/projected/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-kube-api-access-bz6wf\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.265734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.265156 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.265734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.265209 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.265734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.265224 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.265734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.265316 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.265734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.265358 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.265734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.265657 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.267880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.265769 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.267880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.266751 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfcmm\" (UniqueName: \"kubernetes.io/projected/11c5b443-e562-448f-a540-2273f5ae86ca-kube-api-access-cfcmm\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:21.267880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.266775 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c5b443-e562-448f-a540-2273f5ae86ca-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:21.267880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.266791 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:21.267880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.266806 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:21.267880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.266819 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:21.267880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.266834 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:21.267880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.266851 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c5b443-e562-448f-a540-2273f5ae86ca-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:21.267880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.266866 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c5b443-e562-448f-a540-2273f5ae86ca-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:21.267880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.266881 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c5b443-e562-448f-a540-2273f5ae86ca-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:21.267880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.267297 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.267880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.267749 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.268524 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.268504 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.274941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.274889 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.275345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.275026 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz6wf\" (UniqueName: \"kubernetes.io/projected/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-kube-api-access-bz6wf\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.311415 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.311385 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:21.311932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.311908 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:21.312789 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.312766 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:44:21.392084 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.392046 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:21.590361 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.590331 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" Oct 31 03:44:21.590361 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.590347 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx" event={"ID":"11c5b443-e562-448f-a540-2273f5ae86ca","Type":"ContainerDied","Data":"caeb1d1b14aa361ac55a7cdb2ff4eabf8ab735b64b3a6b9deef27a2498a581e2"} Oct 31 03:44:21.590958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.590417 5490 scope.go:122] "RemoveContainer" containerID="d37131b9f898f21b1425d43addcf6b69f1590f3344ea08bb6a598180a05fdf32" Oct 31 03:44:21.613574 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.613523 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx"] Oct 31 03:44:21.618882 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.618851 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-9wbqx"] Oct 31 03:44:21.680281 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.680206 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp"] Oct 31 03:44:21.680441 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.680419 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" podUID="b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" containerName="istio-proxy" containerID="cri-o://602aee522b4370dd4d6334ca9f0a1807409b409dfd015cf3fef751036df1003f" gracePeriod=30 Oct 31 03:44:21.720254 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.720214 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:44:21.720395 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.720301 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:44:21.720395 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.720351 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:44:21.725403 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.725371 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s"] Oct 31 03:44:21.811313 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:21.805799 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:22.600748 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:22.600706 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" event={"ID":"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f","Type":"ContainerStarted","Data":"2d93e43a037cd18f219826a1027f234e43eb8c6e6b81cf3c4cbfdbc0fdf573fe"} Oct 31 03:44:22.601219 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:22.600765 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" event={"ID":"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f","Type":"ContainerStarted","Data":"bb652916a26ab7283f797531401bc4c1ff940163c1b7356dc8dc03c0cbe8a1ff"} Oct 31 03:44:22.632700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:22.632349 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" podStartSLOduration=40.632325115 podStartE2EDuration="40.632325115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:22.625591835 +0000 UTC m=+1697.160286791" watchObservedRunningTime="2026-10-31 03:44:22.632325115 +0000 UTC m=+1697.167020071" Oct 31 03:44:22.725713 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:22.723173 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:23.393080 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:23.393046 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:23.394974 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:23.394947 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 03:44:23.395103 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:23.394996 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" podUID="09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 03:44:23.524859 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:23.521323 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:23.588930 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:23.588882 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c5b443-e562-448f-a540-2273f5ae86ca" path="/var/lib/kubelet/pods/11c5b443-e562-448f-a540-2273f5ae86ca/volumes" Oct 31 03:44:24.188725 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:24.188691 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:24.392995 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:24.392959 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 03:44:24.393148 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:24.393015 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" podUID="09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 03:44:25.105791 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:25.104164 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:25.392553 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:25.392466 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 03:44:25.392964 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:25.392539 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" podUID="09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 03:44:25.926325 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:25.926287 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:26.393087 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:26.393001 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 03:44:26.393087 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:26.393061 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" podUID="09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 03:44:26.797275 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:26.797217 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:27.160709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.160666 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:44:27.209550 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.209468 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8"] Oct 31 03:44:27.216601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.216493 5490 cpu_manager.go:414] "containerMap: removing container" podUID="b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" containerName="istio-proxy" Oct 31 03:44:27.216601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.216531 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" containerName="istio-proxy" Oct 31 03:44:27.216932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.216881 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" containerName="istio-proxy" Oct 31 03:44:27.220718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.220693 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-workload-certs\") pod \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " Oct 31 03:44:27.222266 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.222245 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-data\") pod \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " Oct 31 03:44:27.222790 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.222770 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-podinfo\") pod \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " Oct 31 03:44:27.222978 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.222959 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-envoy\") pod \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " Oct 31 03:44:27.264826 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.222141 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-workload-certs" pod "b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" (UID: "b94f248b-c248-4eb1-ac7c-9d22f5cec9bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:27.264826 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.222667 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-data" pod "b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" (UID: "b94f248b-c248-4eb1-ac7c-9d22f5cec9bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:27.264826 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.263080 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.265329 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.265305 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-workload-socket\") pod \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " Oct 31 03:44:27.266316 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.266284 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-token\" (UniqueName: \"kubernetes.io/projected/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-token\") pod \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " Oct 31 03:44:27.266443 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.266389 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-kube-api-access-4vbt4\" (UniqueName: \"kubernetes.io/projected/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-kube-api-access-4vbt4\") pod \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " Oct 31 03:44:27.266443 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.266429 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-credential-socket\") pod \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " Oct 31 03:44:27.266818 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.266792 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istiod-ca-cert\") pod \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\" (UID: \"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf\") " Oct 31 03:44:27.267768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.267745 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:27.269627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.267907 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:27.269627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.267759 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-podinfo" pod "b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" (UID: "b94f248b-c248-4eb1-ac7c-9d22f5cec9bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:44:27.269627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.268010 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8"] Oct 31 03:44:27.269627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.268064 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-credential-socket" pod "b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" (UID: "b94f248b-c248-4eb1-ac7c-9d22f5cec9bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:27.269627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.268176 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-workload-socket" pod "b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" (UID: "b94f248b-c248-4eb1-ac7c-9d22f5cec9bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:27.269627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.268261 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istiod-ca-cert" pod "b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" (UID: "b94f248b-c248-4eb1-ac7c-9d22f5cec9bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:44:27.272601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.272564 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-envoy" pod "b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" (UID: "b94f248b-c248-4eb1-ac7c-9d22f5cec9bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:27.274730 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.274624 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-token" pod "b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" (UID: "b94f248b-c248-4eb1-ac7c-9d22f5cec9bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:27.275641 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.275614 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-kube-api-access-4vbt4" pod "b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" (UID: "b94f248b-c248-4eb1-ac7c-9d22f5cec9bf"). InnerVolumeSpecName "kube-api-access-4vbt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:27.369867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.369824 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.370089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.369873 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.370089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.369994 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.370089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.370074 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/101e77ad-e8f5-4222-83ee-982ac3954dcc-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.370292 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.370272 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7z8h\" (UniqueName: \"kubernetes.io/projected/101e77ad-e8f5-4222-83ee-982ac3954dcc-kube-api-access-b7z8h\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.370390 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.370307 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.370434 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.370422 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.370505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.370494 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.370589 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.370577 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.370755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.370743 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:27.370788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.370757 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:27.370788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.370768 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vbt4\" (UniqueName: \"kubernetes.io/projected/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-kube-api-access-4vbt4\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:27.370788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.370778 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:27.370872 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.370791 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:27.370872 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.370800 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:27.370872 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.370809 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:27.392575 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.392544 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body= Oct 31 03:44:27.392729 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.392598 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" podUID="09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" Oct 31 03:44:27.472457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.472369 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7z8h\" (UniqueName: \"kubernetes.io/projected/101e77ad-e8f5-4222-83ee-982ac3954dcc-kube-api-access-b7z8h\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.472457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.472438 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.472963 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.472498 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.472963 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.472563 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.472963 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.472671 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.473177 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.473151 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.473240 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.473200 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.473301 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.473283 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.473355 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.473322 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.473407 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.473390 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.473490 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.473473 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/101e77ad-e8f5-4222-83ee-982ac3954dcc-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.474325 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.474298 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/101e77ad-e8f5-4222-83ee-982ac3954dcc-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.475813 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.475789 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.476035 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.476007 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.476114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.476051 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.479073 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.479047 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.482774 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.482749 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7z8h\" (UniqueName: \"kubernetes.io/projected/101e77ad-e8f5-4222-83ee-982ac3954dcc-kube-api-access-b7z8h\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.483887 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.483862 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.599277 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.599245 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:27.669702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.668795 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:27.783188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.783129 5490 generic.go:356] "Generic (PLEG): container finished" podID="b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" containerID="602aee522b4370dd4d6334ca9f0a1807409b409dfd015cf3fef751036df1003f" exitCode=0 Oct 31 03:44:27.783300 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.783241 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" Oct 31 03:44:27.784980 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.783868 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" event={"ID":"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf","Type":"ContainerDied","Data":"602aee522b4370dd4d6334ca9f0a1807409b409dfd015cf3fef751036df1003f"} Oct 31 03:44:27.784980 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.783925 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp" event={"ID":"b94f248b-c248-4eb1-ac7c-9d22f5cec9bf","Type":"ContainerDied","Data":"d8292039bad2763264aeaab7c64d493ec90922a0c149d621d2f09a482ded66d0"} Oct 31 03:44:27.784980 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.783957 5490 scope.go:122] "RemoveContainer" containerID="602aee522b4370dd4d6334ca9f0a1807409b409dfd015cf3fef751036df1003f" Oct 31 03:44:27.877767 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.864508 5490 scope.go:122] "RemoveContainer" containerID="602aee522b4370dd4d6334ca9f0a1807409b409dfd015cf3fef751036df1003f" Oct 31 03:44:27.882958 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:44:27.882922 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"602aee522b4370dd4d6334ca9f0a1807409b409dfd015cf3fef751036df1003f\": container with ID starting with 602aee522b4370dd4d6334ca9f0a1807409b409dfd015cf3fef751036df1003f not found: ID does not exist" containerID="602aee522b4370dd4d6334ca9f0a1807409b409dfd015cf3fef751036df1003f" Oct 31 03:44:27.883077 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.882976 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"602aee522b4370dd4d6334ca9f0a1807409b409dfd015cf3fef751036df1003f"} err="failed to get container status \"602aee522b4370dd4d6334ca9f0a1807409b409dfd015cf3fef751036df1003f\": rpc error: code = NotFound desc = could not find container \"602aee522b4370dd4d6334ca9f0a1807409b409dfd015cf3fef751036df1003f\": container with ID starting with 602aee522b4370dd4d6334ca9f0a1807409b409dfd015cf3fef751036df1003f not found: ID does not exist" Oct 31 03:44:27.900623 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.900382 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp"] Oct 31 03:44:27.911742 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:27.909761 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-rbhdp"] Oct 31 03:44:28.109252 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:28.109218 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8"] Oct 31 03:44:28.109252 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:44:28.109242 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101e77ad_e8f5_4222_83ee_982ac3954dcc.slice/crio-8d675da5765af158d1b595feb0498851bfdff08afdd7d210809b009bb8363786 WatchSource:0}: Error finding container 8d675da5765af158d1b595feb0498851bfdff08afdd7d210809b009bb8363786: Status 404 returned error can't find the container with id 8d675da5765af158d1b595feb0498851bfdff08afdd7d210809b009bb8363786 Oct 31 03:44:28.112772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:28.112735 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:44:28.112893 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:28.112806 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:44:28.112893 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:28.112836 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:44:28.415711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:28.413944 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:28.560474 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:28.558867 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:28.560474 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:28.560241 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:28.562246 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:28.561600 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:44:28.795933 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:28.795831 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" event={"ID":"101e77ad-e8f5-4222-83ee-982ac3954dcc","Type":"ContainerStarted","Data":"64382e4e17f65f1d673c6d45af5f19a58840ec3a10e32b2caf5262d45b8430c2"} Oct 31 03:44:28.795933 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:28.795897 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" event={"ID":"101e77ad-e8f5-4222-83ee-982ac3954dcc","Type":"ContainerStarted","Data":"8d675da5765af158d1b595feb0498851bfdff08afdd7d210809b009bb8363786"} Oct 31 03:44:28.857547 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:28.857489 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" podStartSLOduration=67.857472936 podStartE2EDuration="1m7.857472936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:28.85590472 +0000 UTC m=+1703.390599699" watchObservedRunningTime="2026-10-31 03:44:28.857472936 +0000 UTC m=+1703.392167891" Oct 31 03:44:29.182667 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:29.182372 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:29.579034 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:29.578956 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b94f248b-c248-4eb1-ac7c-9d22f5cec9bf" path="/var/lib/kubelet/pods/b94f248b-c248-4eb1-ac7c-9d22f5cec9bf/volumes" Oct 31 03:44:29.677594 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:29.677556 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:29.679758 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:29.679730 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:44:29.679897 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:29.679783 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" podUID="101e77ad-e8f5-4222-83ee-982ac3954dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:44:29.878398 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:29.878308 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:30.619459 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:30.619424 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:30.678086 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:30.677065 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:44:30.678086 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:30.677131 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" podUID="101e77ad-e8f5-4222-83ee-982ac3954dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:44:31.380800 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:31.380763 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:31.677439 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:31.677400 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:44:31.677935 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:31.677464 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" podUID="101e77ad-e8f5-4222-83ee-982ac3954dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:44:32.157522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:32.157316 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:32.678153 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:32.678112 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:44:32.678565 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:32.678200 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" podUID="101e77ad-e8f5-4222-83ee-982ac3954dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:44:32.902706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:32.902648 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:33.677845 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:33.677813 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:44:33.678040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:33.677874 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" podUID="101e77ad-e8f5-4222-83ee-982ac3954dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:44:33.728705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:33.716914 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:34.380458 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:34.380415 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:34.677649 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:34.677599 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 31 03:44:34.677897 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:34.677746 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" podUID="101e77ad-e8f5-4222-83ee-982ac3954dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 31 03:44:35.113745 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:35.113637 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:35.683603 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:35.683561 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:35.684061 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:35.684035 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:35.684900 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:35.684876 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:44:35.775027 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:35.774984 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:36.440203 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:36.440136 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:37.217345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:37.217296 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:37.886550 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:37.886463 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:38.521212 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:38.521172 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:38.676049 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:38.673768 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:39.379025 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:39.378984 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:39.985903 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:39.985853 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:40.682020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:40.681978 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:41.614227 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:41.614184 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:42.300735 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:42.295218 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:42.906445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:42.906398 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:43.572724 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:43.572663 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:44.218337 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:44.218301 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:44.868807 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:44.863951 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:45.526331 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:45.526289 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:46.225449 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:46.225412 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:46.885558 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:46.885303 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:47.133856 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:44:47.133818 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:44:47.570941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:47.570906 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:48.260313 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:48.260276 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:48.885343 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:48.885302 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:49.733359 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:49.733322 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:50.394739 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:50.394697 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:51.083293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:51.083259 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:51.250453 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:51.241649 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s"] Oct 31 03:44:51.250453 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:51.242097 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" podUID="35b39830-7aed-419d-b82f-fbf9c32128fa" containerName="istio-proxy" containerID="cri-o://a6f7cb0b21e695fa4ba2d3e897c29b29f2eec10e3f9950815dfb298278c04f89" gracePeriod=30 Oct 31 03:44:51.880624 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:51.880574 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:52.696115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:52.696077 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:53.300700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:53.298017 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:54.091702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:54.091647 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:54.685034 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:54.684995 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:55.314386 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:55.314344 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:56.081346 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.081311 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:56.575224 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.575197 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:44:56.625104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.624963 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz"] Oct 31 03:44:56.626799 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.626773 5490 cpu_manager.go:414] "containerMap: removing container" podUID="35b39830-7aed-419d-b82f-fbf9c32128fa" containerName="istio-proxy" Oct 31 03:44:56.626986 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.626970 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35b39830-7aed-419d-b82f-fbf9c32128fa" containerName="istio-proxy" Oct 31 03:44:56.627318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.627299 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="35b39830-7aed-419d-b82f-fbf9c32128fa" containerName="istio-proxy" Oct 31 03:44:56.634330 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.634309 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz"] Oct 31 03:44:56.634558 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.634419 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.664405 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.663920 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-data\") pod \"35b39830-7aed-419d-b82f-fbf9c32128fa\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " Oct 31 03:44:56.664405 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.664030 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b39830-7aed-419d-b82f-fbf9c32128fa-kube-api-access-mvcll\" (UniqueName: \"kubernetes.io/projected/35b39830-7aed-419d-b82f-fbf9c32128fa-kube-api-access-mvcll\") pod \"35b39830-7aed-419d-b82f-fbf9c32128fa\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " Oct 31 03:44:56.664405 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.664269 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-data" pod "35b39830-7aed-419d-b82f-fbf9c32128fa" (UID: "35b39830-7aed-419d-b82f-fbf9c32128fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:56.664405 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.664347 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35b39830-7aed-419d-b82f-fbf9c32128fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35b39830-7aed-419d-b82f-fbf9c32128fa-istiod-ca-cert\") pod \"35b39830-7aed-419d-b82f-fbf9c32128fa\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " Oct 31 03:44:56.664795 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.664450 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-workload-certs\") pod \"35b39830-7aed-419d-b82f-fbf9c32128fa\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " Oct 31 03:44:56.664795 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.664508 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-podinfo\") pod \"35b39830-7aed-419d-b82f-fbf9c32128fa\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " Oct 31 03:44:56.665209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.664654 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b39830-7aed-419d-b82f-fbf9c32128fa-istiod-ca-cert" pod "35b39830-7aed-419d-b82f-fbf9c32128fa" (UID: "35b39830-7aed-419d-b82f-fbf9c32128fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:44:56.665209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.664808 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-workload-certs" pod "35b39830-7aed-419d-b82f-fbf9c32128fa" (UID: "35b39830-7aed-419d-b82f-fbf9c32128fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:56.665209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.664753 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-token\" (UniqueName: \"kubernetes.io/projected/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-token\") pod \"35b39830-7aed-419d-b82f-fbf9c32128fa\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " Oct 31 03:44:56.665452 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.665249 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-workload-socket\") pod \"35b39830-7aed-419d-b82f-fbf9c32128fa\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " Oct 31 03:44:56.665452 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.665286 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-envoy\") pod \"35b39830-7aed-419d-b82f-fbf9c32128fa\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " Oct 31 03:44:56.665452 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.665386 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-credential-socket\") pod \"35b39830-7aed-419d-b82f-fbf9c32128fa\" (UID: \"35b39830-7aed-419d-b82f-fbf9c32128fa\") " Oct 31 03:44:56.666952 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.666734 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35b39830-7aed-419d-b82f-fbf9c32128fa-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:56.666952 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.666757 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:56.666952 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.666772 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:56.666952 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.666837 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-credential-socket" pod "35b39830-7aed-419d-b82f-fbf9c32128fa" (UID: "35b39830-7aed-419d-b82f-fbf9c32128fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:56.667322 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.667301 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-workload-socket" pod "35b39830-7aed-419d-b82f-fbf9c32128fa" (UID: "35b39830-7aed-419d-b82f-fbf9c32128fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:56.668187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.668160 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b39830-7aed-419d-b82f-fbf9c32128fa-kube-api-access-mvcll" pod "35b39830-7aed-419d-b82f-fbf9c32128fa" (UID: "35b39830-7aed-419d-b82f-fbf9c32128fa"). InnerVolumeSpecName "kube-api-access-mvcll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:56.668772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.668737 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-envoy" pod "35b39830-7aed-419d-b82f-fbf9c32128fa" (UID: "35b39830-7aed-419d-b82f-fbf9c32128fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:44:56.675160 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.675117 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-token" pod "35b39830-7aed-419d-b82f-fbf9c32128fa" (UID: "35b39830-7aed-419d-b82f-fbf9c32128fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:44:56.676542 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.676515 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-podinfo" pod "35b39830-7aed-419d-b82f-fbf9c32128fa" (UID: "35b39830-7aed-419d-b82f-fbf9c32128fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:44:56.773039 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.771037 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:56.773039 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.772866 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjl9q\" (UniqueName: \"kubernetes.io/projected/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-kube-api-access-xjl9q\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.773039 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.772930 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-workload-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.773039 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.772975 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istiod-ca-cert\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.773425 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.773165 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-token\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.773425 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.773231 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-workload-certs\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.773540 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.773436 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-envoy\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.773540 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.773477 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-podinfo\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.773647 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.773580 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-data\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.773751 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.773723 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-credential-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.773902 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.773884 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:56.773976 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.773904 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:56.773976 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.773914 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:56.773976 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.773925 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35b39830-7aed-419d-b82f-fbf9c32128fa-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:56.773976 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.773936 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvcll\" (UniqueName: \"kubernetes.io/projected/35b39830-7aed-419d-b82f-fbf9c32128fa-kube-api-access-mvcll\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:56.773976 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.773953 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35b39830-7aed-419d-b82f-fbf9c32128fa-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:44:56.876076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.875978 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-workload-certs\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.876254 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.876113 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-envoy\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.876254 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.876134 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-podinfo\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.876254 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.876173 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-data\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.876456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.876259 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-credential-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.876456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.876377 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjl9q\" (UniqueName: \"kubernetes.io/projected/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-kube-api-access-xjl9q\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.876456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.876421 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-workload-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.876456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.876430 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-workload-certs\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.876456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.876453 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istiod-ca-cert\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.876816 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.876606 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-data\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.876816 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.876632 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-token\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.876816 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.876721 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-credential-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.876986 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.876893 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-workload-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.877232 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.877201 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istiod-ca-cert\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.878604 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.878581 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-envoy\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.878894 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.878876 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-podinfo\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.885936 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.885914 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-token\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.886247 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.886224 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjl9q\" (UniqueName: \"kubernetes.io/projected/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-kube-api-access-xjl9q\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:56.955460 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:56.955425 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:57.124936 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:57.124898 5490 generic.go:356] "Generic (PLEG): container finished" podID="35b39830-7aed-419d-b82f-fbf9c32128fa" containerID="a6f7cb0b21e695fa4ba2d3e897c29b29f2eec10e3f9950815dfb298278c04f89" exitCode=0 Oct 31 03:44:57.125417 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:57.124994 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" Oct 31 03:44:57.125417 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:57.125032 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" event={"ID":"35b39830-7aed-419d-b82f-fbf9c32128fa","Type":"ContainerDied","Data":"a6f7cb0b21e695fa4ba2d3e897c29b29f2eec10e3f9950815dfb298278c04f89"} Oct 31 03:44:57.125417 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:57.125200 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s" event={"ID":"35b39830-7aed-419d-b82f-fbf9c32128fa","Type":"ContainerDied","Data":"4734d76071dc676eb9379ce3a8f4c64caf9815776fb9d54c360269bbac405665"} Oct 31 03:44:57.125417 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:57.125245 5490 scope.go:122] "RemoveContainer" containerID="a6f7cb0b21e695fa4ba2d3e897c29b29f2eec10e3f9950815dfb298278c04f89" Oct 31 03:44:57.140815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:57.140626 5490 scope.go:122] "RemoveContainer" containerID="a6f7cb0b21e695fa4ba2d3e897c29b29f2eec10e3f9950815dfb298278c04f89" Oct 31 03:44:57.141074 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:44:57.141050 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6f7cb0b21e695fa4ba2d3e897c29b29f2eec10e3f9950815dfb298278c04f89\": container with ID starting with a6f7cb0b21e695fa4ba2d3e897c29b29f2eec10e3f9950815dfb298278c04f89 not found: ID does not exist" containerID="a6f7cb0b21e695fa4ba2d3e897c29b29f2eec10e3f9950815dfb298278c04f89" Oct 31 03:44:57.141165 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:57.141092 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f7cb0b21e695fa4ba2d3e897c29b29f2eec10e3f9950815dfb298278c04f89"} err="failed to get container status \"a6f7cb0b21e695fa4ba2d3e897c29b29f2eec10e3f9950815dfb298278c04f89\": rpc error: code = NotFound desc = could not find container \"a6f7cb0b21e695fa4ba2d3e897c29b29f2eec10e3f9950815dfb298278c04f89\": container with ID starting with a6f7cb0b21e695fa4ba2d3e897c29b29f2eec10e3f9950815dfb298278c04f89 not found: ID does not exist" Oct 31 03:44:57.158533 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:57.158498 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s"] Oct 31 03:44:57.166364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:57.166331 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-zsg7s"] Oct 31 03:44:57.253622 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:57.253592 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz"] Oct 31 03:44:57.254452 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:44:57.254425 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4dcf84_a219_4ff2_b4ff_5549ef2fafe5.slice/crio-86d75959bc8a9bc980cb0da80bbe2b79ab98586affa76142c70b4ebdf78aae69 WatchSource:0}: Error finding container 86d75959bc8a9bc980cb0da80bbe2b79ab98586affa76142c70b4ebdf78aae69: Status 404 returned error can't find the container with id 86d75959bc8a9bc980cb0da80bbe2b79ab98586affa76142c70b4ebdf78aae69 Oct 31 03:44:57.258779 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:57.258746 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:44:57.258895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:57.258853 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:44:57.258962 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:57.258903 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:44:57.428310 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:57.426671 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:57.581386 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:57.581349 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35b39830-7aed-419d-b82f-fbf9c32128fa" path="/var/lib/kubelet/pods/35b39830-7aed-419d-b82f-fbf9c32128fa/volumes" Oct 31 03:44:58.107118 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:58.107061 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:58.165898 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:58.165843 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" event={"ID":"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5","Type":"ContainerStarted","Data":"42f35106e7938dfa12cfa21f82be9b5d6f83663d750e29e61ee0d7330a17df1f"} Oct 31 03:44:58.166481 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:58.165968 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" event={"ID":"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5","Type":"ContainerStarted","Data":"86d75959bc8a9bc980cb0da80bbe2b79ab98586affa76142c70b4ebdf78aae69"} Oct 31 03:44:58.189604 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:58.189537 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" podStartSLOduration=25.189516053 podStartE2EDuration="25.189516053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:58.18904769 +0000 UTC m=+1732.723742658" watchObservedRunningTime="2026-10-31 03:44:58.189516053 +0000 UTC m=+1732.724211009" Oct 31 03:44:58.915529 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:58.915486 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:58.960087 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:58.960051 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:44:58.960498 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:58.960472 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:44:58.960705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:58.960658 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" podUID="db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:44:59.520193 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:59.519254 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:44:59.956299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:59.956259 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:44:59.956528 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:44:59.956323 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" podUID="db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:45:00.079738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:00.079651 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:00.260940 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:00.260843 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:00.408281 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:00.408242 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:00.781813 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:00.781770 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:00.925671 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:00.925629 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:00.956410 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:00.956372 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:45:00.956572 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:00.956438 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" podUID="db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:45:01.116766 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:01.116401 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:01.956817 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:01.956770 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:45:01.957289 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:01.956837 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" podUID="db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:45:01.998704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:01.995984 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:02.865256 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:02.865188 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:02.958701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:02.957909 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:45:02.958701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:02.957987 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" podUID="db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:45:03.617577 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:03.617536 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:03.956154 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:03.956112 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 31 03:45:03.956336 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:03.956196 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" podUID="db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 31 03:45:04.277265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:04.277170 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:04.961181 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:04.961151 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:45:04.961661 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:04.961636 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:45:04.962381 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:04.962359 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:45:04.986356 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:04.986322 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:05.678736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:05.678692 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:06.296702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:06.293782 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:06.940722 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:06.940666 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:07.589636 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:07.589597 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:08.179037 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:08.179002 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:08.870456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:08.870416 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:09.496348 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:09.496298 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:10.102581 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:10.100445 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:10.704194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:10.704157 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:11.299874 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:11.299834 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:12.589006 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:12.588940 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:13.198830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:13.198788 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:13.844586 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:13.844540 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:14.528140 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:14.528101 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:15.218701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:15.217241 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:15.902854 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:15.902799 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:16.698995 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:16.697931 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:17.149648 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:45:17.149522 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:45:17.410024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:17.409970 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:17.548015 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:17.547971 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:18.570697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:18.559242 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:19.694698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:19.683611 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:21.240954 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:21.240884 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:21.652960 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:21.652859 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:22.635807 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:22.629511 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:23.317658 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:23.317621 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:23.985288 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:23.985250 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:24.177547 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.176293 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm"] Oct 31 03:45:24.190963 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.190932 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm"] Oct 31 03:45:24.191314 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.191293 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:24.196430 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.196403 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-50b5f2\"/\"default-dockercfg-kp6sc\"" Oct 31 03:45:24.197733 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.197709 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-50b5f2\"/\"openshift-service-ca.crt\"" Oct 31 03:45:24.198088 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.198069 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-50b5f2\"/\"test-model-50b5f2-kserve-self-signed-certs\"" Oct 31 03:45:24.198515 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.198490 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-50b5f2\"/\"kube-root-ca.crt\"" Oct 31 03:45:24.234425 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.234389 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:24.320989 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.320888 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-dshm\") pod \"test-model-50b5f2-kserve-978bd677f-2clcm\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:24.321180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.321022 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-home\") pod \"test-model-50b5f2-kserve-978bd677f-2clcm\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:24.321180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.321151 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jctxp\" (UniqueName: \"kubernetes.io/projected/7880b1c4-cc27-4692-ae95-5682724c2f4c-kube-api-access-jctxp\") pod \"test-model-50b5f2-kserve-978bd677f-2clcm\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:24.321332 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.321204 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-model-cache\") pod \"test-model-50b5f2-kserve-978bd677f-2clcm\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:24.321332 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.321229 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-tmp-dir\") pod \"test-model-50b5f2-kserve-978bd677f-2clcm\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:24.321454 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.321396 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7880b1c4-cc27-4692-ae95-5682724c2f4c-tls-certs\") pod \"test-model-50b5f2-kserve-978bd677f-2clcm\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:24.431698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.426056 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-home\") pod \"test-model-50b5f2-kserve-978bd677f-2clcm\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:24.431698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.426225 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jctxp\" (UniqueName: \"kubernetes.io/projected/7880b1c4-cc27-4692-ae95-5682724c2f4c-kube-api-access-jctxp\") pod \"test-model-50b5f2-kserve-978bd677f-2clcm\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:24.431698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.426288 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-model-cache\") pod \"test-model-50b5f2-kserve-978bd677f-2clcm\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:24.431698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.426325 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-tmp-dir\") pod \"test-model-50b5f2-kserve-978bd677f-2clcm\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:24.431698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.426470 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7880b1c4-cc27-4692-ae95-5682724c2f4c-tls-certs\") pod \"test-model-50b5f2-kserve-978bd677f-2clcm\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:24.431698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.426654 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-dshm\") pod \"test-model-50b5f2-kserve-978bd677f-2clcm\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:24.431698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.429046 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-model-cache\") pod \"test-model-50b5f2-kserve-978bd677f-2clcm\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:24.431698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.429284 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-home\") pod \"test-model-50b5f2-kserve-978bd677f-2clcm\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:24.431698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.429937 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-tmp-dir\") pod \"test-model-50b5f2-kserve-978bd677f-2clcm\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:24.503557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.503520 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jctxp\" (UniqueName: \"kubernetes.io/projected/7880b1c4-cc27-4692-ae95-5682724c2f4c-kube-api-access-jctxp\") pod \"test-model-50b5f2-kserve-978bd677f-2clcm\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:24.504509 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.504467 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7880b1c4-cc27-4692-ae95-5682724c2f4c-tls-certs\") pod \"test-model-50b5f2-kserve-978bd677f-2clcm\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:24.505474 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.505454 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-dshm\") pod \"test-model-50b5f2-kserve-978bd677f-2clcm\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:24.534941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.534895 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:24.900921 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:24.889659 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm"] Oct 31 03:45:25.018381 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:25.012349 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:25.289132 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:25.289097 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" event={"ID":"7880b1c4-cc27-4692-ae95-5682724c2f4c","Type":"ContainerStarted","Data":"a327decf5510d4892a78d5c8b9e4449857b39a1d25950f1a99df9329619c4c38"} Oct 31 03:45:25.289330 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:25.289139 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" event={"ID":"7880b1c4-cc27-4692-ae95-5682724c2f4c","Type":"ContainerStarted","Data":"608cd445bdaea1b94221fd9c1a3b62916c4ec5f5541be75ca03412fb80187a55"} Oct 31 03:45:25.309957 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:25.309899 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podStartSLOduration=1.309881751 podStartE2EDuration="1.309881751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:25.30725458 +0000 UTC m=+1759.841949545" watchObservedRunningTime="2026-10-31 03:45:25.309881751 +0000 UTC m=+1759.844576706" Oct 31 03:45:25.542247 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:25.536801 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:26.784326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:26.784281 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:26.914486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:26.914448 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:27.331724 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:27.331666 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:27.907495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:27.907454 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:28.579607 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:28.579566 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:29.341011 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:29.340956 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:29.985468 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:29.985432 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:30.735703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:30.733137 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:31.493474 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:31.490430 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:32.224723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:32.224660 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:32.961598 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:32.961333 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:33.783058 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:33.783015 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:34.475654 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:34.475587 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:34.535765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:34.535730 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:34.548297 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:34.548271 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:35.220937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:35.220878 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:35.488973 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:35.488890 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:45:35.971646 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:35.971606 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:36.786702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:36.782385 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:37.623565 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:37.623528 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:38.283987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:38.283944 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:38.890366 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:38.890327 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:39.636906 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:39.636857 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:39.948009 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:39.947975 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:40.524863 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:40.524812 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:41.214983 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:41.214948 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:41.900782 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:41.900665 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:42.594248 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:42.594171 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:43.307651 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:43.307605 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:44.116218 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:44.116180 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:44.869905 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:44.869851 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:45.646189 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:45.645995 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:46.305719 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:46.305283 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:46.938176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:46.938132 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:47.152884 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:45:47.152837 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:45:47.629934 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:47.629892 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:48.373524 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:48.373475 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:49.059017 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:49.058969 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:49.659916 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:49.659866 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:50.402988 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:50.402941 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:51.002812 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:51.002773 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:51.329829 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:51.329730 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:51.467157 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:51.467117 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:51.689040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:51.688990 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:52.499735 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:52.498299 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:53.191860 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:53.191815 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:53.896308 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:53.896236 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:54.110237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:54.110202 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:54.240881 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:54.240845 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:54.535482 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:54.534494 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:55.328582 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:55.326370 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:55.951074 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:55.951026 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:56.599529 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:56.599381 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:57.364757 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:57.362352 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:58.310272 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:58.300616 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:58.955075 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:58.955024 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:45:59.796095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:45:59.795844 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:00.492044 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:00.491022 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:01.111425 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:01.111385 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:01.852619 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:01.852573 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:02.453241 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:02.453202 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:02.825167 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:02.825062 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:02.972148 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:02.972108 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:03.111269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:03.111120 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:03.801087 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:03.801014 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:04.381028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:04.380984 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:05.014084 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:05.014046 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:05.802348 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:05.802287 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:06.506935 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:06.506888 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:07.179649 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:07.179611 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:07.774481 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:07.774441 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:08.524047 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:08.522348 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:09.134088 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:09.134042 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:09.639043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:09.638998 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:10.355521 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:10.355477 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:11.083139 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:11.075700 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:11.693059 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:11.693016 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:12.479060 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:12.479012 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:13.193101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:13.193005 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:13.913160 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:13.898801 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:14.618750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:14.618705 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:15.335698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:15.335583 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:15.985488 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:15.985383 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:16.722698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:16.718386 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:17.134604 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:46:17.134507 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:46:17.451487 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:17.451449 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:17.596608 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:17.596568 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:18.133219 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:18.133175 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:18.834664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:18.833973 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:19.434738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:19.434170 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:20.284449 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:20.284340 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:20.797413 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:20.797374 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:21.227455 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:21.227349 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:22.068033 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:22.067964 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:22.703746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:22.703709 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:23.413852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:23.413799 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:24.100698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:24.094959 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:24.756549 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:24.748448 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:25.622113 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:25.622027 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:26.285602 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:26.285550 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:26.888718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:26.888628 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:27.593101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:27.593065 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:27.944808 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:27.944775 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:28.076496 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:28.076463 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:28.315322 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:28.315211 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:29.093616 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:29.083008 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:29.701345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:29.701303 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:30.450323 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:30.450282 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:31.127549 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:31.121658 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:31.432742 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:31.432702 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:31.577504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:31.577451 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:32.187216 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:32.187179 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:32.878230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:32.878187 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:33.501753 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:33.501615 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:34.302714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:34.301086 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:34.955902 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:34.955851 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:36.284700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:36.283970 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.536758 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:36.509089 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/ready\": context deadline exceeded" Oct 31 03:46:36.540702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:36.536495 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:46:36.732818 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:36.732762 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:46:36.733514 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:36.733108 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.733983 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:36.733136 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:46:36.768704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:36.756340 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.813897 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:36.813783 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.829804 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:36.829752 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.890702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:36.889798 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:36.928843 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:36.928781 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:37.233521 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:37.222540 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:37.313070 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:37.313028 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:37.314272 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:37.314239 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:46:37.348700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:37.340296 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:37.425608 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:37.425574 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:46:37.550333 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:37.550199 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:37.552599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:37.552554 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:46:37.560835 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:37.553508 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:46:37.601092 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:37.601035 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:46:37.686510 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:37.686466 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:38.011308 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:38.011265 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7"] Oct 31 03:46:38.018698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:38.014563 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" podUID="428d2eec-1622-4031-9390-905eb9ede375" containerName="maas-api" containerID="cri-o://b4ad916de4d443291279095bf3dd4edfe09a55252f3da5fd8fdb8568a66e5883" gracePeriod=30 Oct 31 03:46:38.179227 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:38.179103 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:38.428804 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:38.420653 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:38.606318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:38.603656 5490 generic.go:356] "Generic (PLEG): container finished" podID="428d2eec-1622-4031-9390-905eb9ede375" containerID="b4ad916de4d443291279095bf3dd4edfe09a55252f3da5fd8fdb8568a66e5883" exitCode=0 Oct 31 03:46:38.606318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:38.605071 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" event={"ID":"428d2eec-1622-4031-9390-905eb9ede375","Type":"ContainerDied","Data":"b4ad916de4d443291279095bf3dd4edfe09a55252f3da5fd8fdb8568a66e5883"} Oct 31 03:46:38.749105 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:38.749009 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:38.867975 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:38.867935 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" Oct 31 03:46:38.911591 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:38.911545 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:38.960845 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:38.960801 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg"] Oct 31 03:46:38.963592 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:38.963553 5490 cpu_manager.go:414] "containerMap: removing container" podUID="428d2eec-1622-4031-9390-905eb9ede375" containerName="maas-api" Oct 31 03:46:38.963839 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:38.963819 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428d2eec-1622-4031-9390-905eb9ede375" containerName="maas-api" Oct 31 03:46:38.964863 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:38.964837 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="428d2eec-1622-4031-9390-905eb9ede375" containerName="maas-api" Oct 31 03:46:38.970928 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:38.970903 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg"] Oct 31 03:46:38.971573 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:38.971236 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:38.988930 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:38.988894 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428d2eec-1622-4031-9390-905eb9ede375-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/428d2eec-1622-4031-9390-905eb9ede375-openshift-service-ca\") pod \"428d2eec-1622-4031-9390-905eb9ede375\" (UID: \"428d2eec-1622-4031-9390-905eb9ede375\") " Oct 31 03:46:38.989603 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:38.989172 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/428d2eec-1622-4031-9390-905eb9ede375-metrics-certs\" (UniqueName: \"kubernetes.io/secret/428d2eec-1622-4031-9390-905eb9ede375-metrics-certs\") pod \"428d2eec-1622-4031-9390-905eb9ede375\" (UID: \"428d2eec-1622-4031-9390-905eb9ede375\") " Oct 31 03:46:38.989603 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:38.989272 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/428d2eec-1622-4031-9390-905eb9ede375-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/428d2eec-1622-4031-9390-905eb9ede375-maas-api-tls\") pod \"428d2eec-1622-4031-9390-905eb9ede375\" (UID: \"428d2eec-1622-4031-9390-905eb9ede375\") " Oct 31 03:46:38.989603 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:38.989332 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428d2eec-1622-4031-9390-905eb9ede375-kube-api-access-qcb4j\" (UniqueName: \"kubernetes.io/projected/428d2eec-1622-4031-9390-905eb9ede375-kube-api-access-qcb4j\") pod \"428d2eec-1622-4031-9390-905eb9ede375\" (UID: \"428d2eec-1622-4031-9390-905eb9ede375\") " Oct 31 03:46:38.995476 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:38.993534 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428d2eec-1622-4031-9390-905eb9ede375-openshift-service-ca" pod "428d2eec-1622-4031-9390-905eb9ede375" (UID: "428d2eec-1622-4031-9390-905eb9ede375"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:39.032806 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.029416 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b68780-openshift-default-dockercfg-7824b\"" Oct 31 03:46:39.044890 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.044828 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428d2eec-1622-4031-9390-905eb9ede375-kube-api-access-qcb4j" pod "428d2eec-1622-4031-9390-905eb9ede375" (UID: "428d2eec-1622-4031-9390-905eb9ede375"). InnerVolumeSpecName "kube-api-access-qcb4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:39.045508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.045458 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/428d2eec-1622-4031-9390-905eb9ede375-metrics-certs" pod "428d2eec-1622-4031-9390-905eb9ede375" (UID: "428d2eec-1622-4031-9390-905eb9ede375"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:46:39.045824 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.045791 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/428d2eec-1622-4031-9390-905eb9ede375-maas-api-tls" pod "428d2eec-1622-4031-9390-905eb9ede375" (UID: "428d2eec-1622-4031-9390-905eb9ede375"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:46:39.100997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.095068 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209cb838-5556-47a9-96ae-e2378ad8c8b5-istiod-ca-cert\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.100997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.095220 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-workload-certs\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.100997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.095318 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-envoy\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.100997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.095508 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tr8z\" (UniqueName: \"kubernetes.io/projected/209cb838-5556-47a9-96ae-e2378ad8c8b5-kube-api-access-7tr8z\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.100997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.095713 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-token\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.100997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.095837 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-workload-socket\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.100997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.095936 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-data\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.100997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.096008 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-credential-socket\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.100997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.096038 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-podinfo\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.100997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.096298 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/428d2eec-1622-4031-9390-905eb9ede375-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:39.100997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.096313 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/428d2eec-1622-4031-9390-905eb9ede375-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:39.100997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.096324 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/428d2eec-1622-4031-9390-905eb9ede375-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:39.100997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.096333 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcb4j\" (UniqueName: \"kubernetes.io/projected/428d2eec-1622-4031-9390-905eb9ede375-kube-api-access-qcb4j\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:39.198034 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.197996 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-token\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.198199 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.198113 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-workload-socket\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.198199 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.198169 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-data\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.198199 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.198191 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-credential-socket\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.199158 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.198217 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-podinfo\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.199158 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.198362 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209cb838-5556-47a9-96ae-e2378ad8c8b5-istiod-ca-cert\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.199158 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.198400 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-workload-certs\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.199158 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.198442 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-envoy\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.199158 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.198552 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-workload-socket\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.199158 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.198560 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tr8z\" (UniqueName: \"kubernetes.io/projected/209cb838-5556-47a9-96ae-e2378ad8c8b5-kube-api-access-7tr8z\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.199158 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.198691 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-data\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.199158 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.198936 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-workload-certs\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.199856 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.199162 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209cb838-5556-47a9-96ae-e2378ad8c8b5-istiod-ca-cert\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.199856 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.199399 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-credential-socket\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.201867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.201835 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-envoy\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.202057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.202036 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-podinfo\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.207130 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.207102 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-token\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.207354 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.207335 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tr8z\" (UniqueName: \"kubernetes.io/projected/209cb838-5556-47a9-96ae-e2378ad8c8b5-kube-api-access-7tr8z\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.307073 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.306977 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:39.330310 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.330200 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:39.619714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.619538 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" Oct 31 03:46:39.619714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.619566 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7" event={"ID":"428d2eec-1622-4031-9390-905eb9ede375","Type":"ContainerDied","Data":"f2ecc56201eff767d77a237ba0e44985cf614424f53a0ca3d096936c4cabfdda"} Oct 31 03:46:39.619714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.619647 5490 scope.go:122] "RemoveContainer" containerID="b4ad916de4d443291279095bf3dd4edfe09a55252f3da5fd8fdb8568a66e5883" Oct 31 03:46:39.666671 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.666632 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7"] Oct 31 03:46:39.676780 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.676745 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8385c5-7cbbc45686-j2qd7"] Oct 31 03:46:39.722021 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.721990 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg"] Oct 31 03:46:39.724830 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:46:39.724791 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209cb838_5556_47a9_96ae_e2378ad8c8b5.slice/crio-9045f4a5ec70a0b9774d36565b6e367240631f7891f76d078371fe92710b168b WatchSource:0}: Error finding container 9045f4a5ec70a0b9774d36565b6e367240631f7891f76d078371fe92710b168b: Status 404 returned error can't find the container with id 9045f4a5ec70a0b9774d36565b6e367240631f7891f76d078371fe92710b168b Oct 31 03:46:39.734350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.734312 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:46:39.734477 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.734409 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:46:39.734477 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.734457 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:46:39.804042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:39.804005 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:40.664380 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:40.664344 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" event={"ID":"209cb838-5556-47a9-96ae-e2378ad8c8b5","Type":"ContainerStarted","Data":"695ae0d222a4ce1f614904f66c785fb60a236e16e77f07fb2a54bde17bcc8031"} Oct 31 03:46:40.664923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:40.664902 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" event={"ID":"209cb838-5556-47a9-96ae-e2378ad8c8b5","Type":"ContainerStarted","Data":"9045f4a5ec70a0b9774d36565b6e367240631f7891f76d078371fe92710b168b"} Oct 31 03:46:40.703309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:40.703251 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" podStartSLOduration=72.703235042 podStartE2EDuration="1m12.703235042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:46:40.700688672 +0000 UTC m=+1835.235383622" watchObservedRunningTime="2026-10-31 03:46:40.703235042 +0000 UTC m=+1835.237929996" Oct 31 03:46:40.747703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:40.739339 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:40.876282 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:40.876232 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:41.025816 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:41.025620 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:41.307224 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:41.307132 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:41.308817 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:41.308782 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 03:46:41.308961 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:41.308858 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" podUID="209cb838-5556-47a9-96ae-e2378ad8c8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 03:46:41.333349 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:41.333310 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:41.468388 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:41.468347 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:41.581221 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:41.581128 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428d2eec-1622-4031-9390-905eb9ede375" path="/var/lib/kubelet/pods/428d2eec-1622-4031-9390-905eb9ede375/volumes" Oct 31 03:46:41.855559 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:41.855435 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:42.307693 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:42.307632 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 03:46:42.307868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:42.307773 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" podUID="209cb838-5556-47a9-96ae-e2378ad8c8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 03:46:42.535700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:42.530807 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:43.307789 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:43.307753 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 03:46:43.308292 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:43.307813 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" podUID="209cb838-5556-47a9-96ae-e2378ad8c8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 03:46:43.389291 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:43.389227 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:44.208769 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:44.208289 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:44.307976 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:44.307943 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 03:46:44.308413 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:44.308010 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" podUID="209cb838-5556-47a9-96ae-e2378ad8c8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 03:46:44.961701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:44.953515 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:45.308037 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:45.307949 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 03:46:45.308428 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:45.308034 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" podUID="209cb838-5556-47a9-96ae-e2378ad8c8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 03:46:45.652990 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:45.652726 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:45.804747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:45.799707 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:45.981315 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:45.981279 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf"] Oct 31 03:46:45.982284 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:45.982248 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" podUID="459ac1eb-d4bf-4b60-badd-07fe1893f3d0" containerName="istio-proxy" containerID="cri-o://21b58edfaf6c7c047e9abe559094ad809945cd1aea40f442e2c758eb6ea9efd8" gracePeriod=30 Oct 31 03:46:45.990613 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:45.990567 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5"] Oct 31 03:46:45.990949 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:45.990920 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" podUID="240a8cbb-b982-4490-9929-ece58bc06d5f" containerName="istio-proxy" containerID="cri-o://6f51f90454ddb1cfcb3b2ca63d7d4a468c72144e80f5b07c30f553266611e729" gracePeriod=30 Oct 31 03:46:46.080736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:46.068009 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:46.308040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:46.307928 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 31 03:46:46.308312 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:46.308282 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" podUID="209cb838-5556-47a9-96ae-e2378ad8c8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 31 03:46:46.331698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:46.331406 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:46.915700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:46.911177 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:47.158532 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:46:47.158492 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:46:47.315118 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:47.315031 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:47.315581 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:47.315400 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:47.316278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:47.316260 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:46:47.794288 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:47.793968 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:48.426317 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:48.426110 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:49.071789 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:49.071298 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:49.766700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:49.766204 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:50.391518 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:50.391484 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:51.086211 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.086152 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:51.418574 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.418542 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:46:51.423085 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.423051 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:46:51.456015 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.455961 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5"] Oct 31 03:46:51.457376 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.457343 5490 cpu_manager.go:414] "containerMap: removing container" podUID="240a8cbb-b982-4490-9929-ece58bc06d5f" containerName="istio-proxy" Oct 31 03:46:51.457376 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.457378 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240a8cbb-b982-4490-9929-ece58bc06d5f" containerName="istio-proxy" Oct 31 03:46:51.457643 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.457415 5490 cpu_manager.go:414] "containerMap: removing container" podUID="459ac1eb-d4bf-4b60-badd-07fe1893f3d0" containerName="istio-proxy" Oct 31 03:46:51.457643 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.457424 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459ac1eb-d4bf-4b60-badd-07fe1893f3d0" containerName="istio-proxy" Oct 31 03:46:51.457812 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.457663 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="459ac1eb-d4bf-4b60-badd-07fe1893f3d0" containerName="istio-proxy" Oct 31 03:46:51.457812 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.457725 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="240a8cbb-b982-4490-9929-ece58bc06d5f" containerName="istio-proxy" Oct 31 03:46:51.475118 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.475084 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.481812 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.480355 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn"] Oct 31 03:46:51.496583 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.496539 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5"] Oct 31 03:46:51.496773 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.496596 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn"] Oct 31 03:46:51.496773 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.496730 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.505903 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.505860 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b81d65-openshift-default-dockercfg-58kth\"" Oct 31 03:46:51.520867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.520789 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-kube-api-access-ggvln\" (UniqueName: \"kubernetes.io/projected/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-kube-api-access-ggvln\") pod \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " Oct 31 03:46:51.520867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.520843 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-workload-certs\") pod \"240a8cbb-b982-4490-9929-ece58bc06d5f\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " Oct 31 03:46:51.521107 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.520903 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/240a8cbb-b982-4490-9929-ece58bc06d5f-kube-api-access-nlbln\" (UniqueName: \"kubernetes.io/projected/240a8cbb-b982-4490-9929-ece58bc06d5f-kube-api-access-nlbln\") pod \"240a8cbb-b982-4490-9929-ece58bc06d5f\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " Oct 31 03:46:51.521107 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.520930 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-workload-socket\") pod \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " Oct 31 03:46:51.521107 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.521027 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-token\" (UniqueName: \"kubernetes.io/projected/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-token\") pod \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " Oct 31 03:46:51.522445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.521123 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-data\") pod \"240a8cbb-b982-4490-9929-ece58bc06d5f\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " Oct 31 03:46:51.522445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.521174 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-envoy\") pod \"240a8cbb-b982-4490-9929-ece58bc06d5f\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " Oct 31 03:46:51.522445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.521264 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-podinfo\") pod \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " Oct 31 03:46:51.522445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.521411 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-data\") pod \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " Oct 31 03:46:51.522445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.521499 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/240a8cbb-b982-4490-9929-ece58bc06d5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/240a8cbb-b982-4490-9929-ece58bc06d5f-istiod-ca-cert\") pod \"240a8cbb-b982-4490-9929-ece58bc06d5f\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " Oct 31 03:46:51.522445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.521536 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-credential-socket\") pod \"240a8cbb-b982-4490-9929-ece58bc06d5f\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " Oct 31 03:46:51.522445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.521589 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-workload-socket\") pod \"240a8cbb-b982-4490-9929-ece58bc06d5f\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " Oct 31 03:46:51.522445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.521671 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-token\" (UniqueName: \"kubernetes.io/projected/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-token\") pod \"240a8cbb-b982-4490-9929-ece58bc06d5f\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " Oct 31 03:46:51.522445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.521716 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-credential-socket\") pod \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " Oct 31 03:46:51.522445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.521760 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istiod-ca-cert\") pod \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " Oct 31 03:46:51.522445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.521851 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-workload-certs\") pod \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " Oct 31 03:46:51.522445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.521886 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-podinfo\") pod \"240a8cbb-b982-4490-9929-ece58bc06d5f\" (UID: \"240a8cbb-b982-4490-9929-ece58bc06d5f\") " Oct 31 03:46:51.522445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.522002 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-envoy\") pod \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\" (UID: \"459ac1eb-d4bf-4b60-badd-07fe1893f3d0\") " Oct 31 03:46:51.522445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.522338 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-workload-socket\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.522445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.522426 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljn8x\" (UniqueName: \"kubernetes.io/projected/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-kube-api-access-ljn8x\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.525937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.522489 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-token\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.525937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.522538 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istiod-ca-cert\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.525937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.522567 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-envoy\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.525937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.522650 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-data\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.525937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.522723 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-data" pod "240a8cbb-b982-4490-9929-ece58bc06d5f" (UID: "240a8cbb-b982-4490-9929-ece58bc06d5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.525937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.522837 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-credential-socket\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.525937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.522962 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-podinfo\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.525937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.523156 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-workload-certs\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.525937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.523263 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-workload-certs" pod "240a8cbb-b982-4490-9929-ece58bc06d5f" (UID: "240a8cbb-b982-4490-9929-ece58bc06d5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.525937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.523287 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.525937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.523863 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-data" pod "459ac1eb-d4bf-4b60-badd-07fe1893f3d0" (UID: "459ac1eb-d4bf-4b60-badd-07fe1893f3d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.525937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.524956 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-workload-socket" pod "459ac1eb-d4bf-4b60-badd-07fe1893f3d0" (UID: "459ac1eb-d4bf-4b60-badd-07fe1893f3d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.525937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.524987 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/240a8cbb-b982-4490-9929-ece58bc06d5f-istiod-ca-cert" pod "240a8cbb-b982-4490-9929-ece58bc06d5f" (UID: "240a8cbb-b982-4490-9929-ece58bc06d5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:51.525937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.525568 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-credential-socket" pod "240a8cbb-b982-4490-9929-ece58bc06d5f" (UID: "240a8cbb-b982-4490-9929-ece58bc06d5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.536048 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.535914 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-envoy" pod "240a8cbb-b982-4490-9929-ece58bc06d5f" (UID: "240a8cbb-b982-4490-9929-ece58bc06d5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.536737 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.536469 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-token" pod "459ac1eb-d4bf-4b60-badd-07fe1893f3d0" (UID: "459ac1eb-d4bf-4b60-badd-07fe1893f3d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:51.536737 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.536597 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-workload-socket" pod "240a8cbb-b982-4490-9929-ece58bc06d5f" (UID: "240a8cbb-b982-4490-9929-ece58bc06d5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.536737 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.536613 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-podinfo" pod "240a8cbb-b982-4490-9929-ece58bc06d5f" (UID: "240a8cbb-b982-4490-9929-ece58bc06d5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:46:51.537668 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.537303 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/240a8cbb-b982-4490-9929-ece58bc06d5f-kube-api-access-nlbln" pod "240a8cbb-b982-4490-9929-ece58bc06d5f" (UID: "240a8cbb-b982-4490-9929-ece58bc06d5f"). InnerVolumeSpecName "kube-api-access-nlbln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:51.538543 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.538506 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-envoy" pod "459ac1eb-d4bf-4b60-badd-07fe1893f3d0" (UID: "459ac1eb-d4bf-4b60-badd-07fe1893f3d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.538634 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.538593 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-kube-api-access-ggvln" pod "459ac1eb-d4bf-4b60-badd-07fe1893f3d0" (UID: "459ac1eb-d4bf-4b60-badd-07fe1893f3d0"). InnerVolumeSpecName "kube-api-access-ggvln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:51.539492 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.539457 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-workload-certs" pod "459ac1eb-d4bf-4b60-badd-07fe1893f3d0" (UID: "459ac1eb-d4bf-4b60-badd-07fe1893f3d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.539988 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.539952 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istiod-ca-cert" pod "459ac1eb-d4bf-4b60-badd-07fe1893f3d0" (UID: "459ac1eb-d4bf-4b60-badd-07fe1893f3d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:46:51.540160 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.540131 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-credential-socket" pod "459ac1eb-d4bf-4b60-badd-07fe1893f3d0" (UID: "459ac1eb-d4bf-4b60-badd-07fe1893f3d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:46:51.542790 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.542763 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-token" pod "240a8cbb-b982-4490-9929-ece58bc06d5f" (UID: "240a8cbb-b982-4490-9929-ece58bc06d5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:46:51.550780 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.550749 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-podinfo" pod "459ac1eb-d4bf-4b60-badd-07fe1893f3d0" (UID: "459ac1eb-d4bf-4b60-badd-07fe1893f3d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:46:51.624723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.624643 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-workload-certs\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.624905 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.624834 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-workload-socket\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.624905 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.624888 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/015afcb4-d156-4163-a342-f84471c9dd51-istio-token\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.625123 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.624928 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljn8x\" (UniqueName: \"kubernetes.io/projected/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-kube-api-access-ljn8x\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.625123 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.624988 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-workload-socket\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.625123 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.625016 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/015afcb4-d156-4163-a342-f84471c9dd51-istio-podinfo\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.625123 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.625050 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-token\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.625123 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.625072 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-workload-certs\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.625123 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.625080 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-istio-envoy\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.625424 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.625137 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-workload-socket\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.625424 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.625217 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istiod-ca-cert\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.626269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.625449 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-envoy\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.626269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.625598 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-data\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.626269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.625653 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-credential-socket\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.626269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.625724 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kr8z\" (UniqueName: \"kubernetes.io/projected/015afcb4-d156-4163-a342-f84471c9dd51-kube-api-access-6kr8z\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.626269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.625896 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-podinfo\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.626269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.625957 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-istio-data\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.626269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626012 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-credential-socket\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.626269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626108 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/015afcb4-d156-4163-a342-f84471c9dd51-istiod-ca-cert\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.626269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626241 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-workload-certs\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.626788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626420 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.626788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626441 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggvln\" (UniqueName: \"kubernetes.io/projected/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-kube-api-access-ggvln\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.626788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626460 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.626788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626475 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlbln\" (UniqueName: \"kubernetes.io/projected/240a8cbb-b982-4490-9929-ece58bc06d5f-kube-api-access-nlbln\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.626788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626491 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.626788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626506 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.626788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626522 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.626788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626540 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.626788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626555 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.626788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626571 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/240a8cbb-b982-4490-9929-ece58bc06d5f-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.626788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626585 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.626788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626602 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/240a8cbb-b982-4490-9929-ece58bc06d5f-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.626788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626616 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.626788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626631 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.626788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626646 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.626788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626661 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/459ac1eb-d4bf-4b60-badd-07fe1893f3d0-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.626788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626693 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/240a8cbb-b982-4490-9929-ece58bc06d5f-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:46:51.631298 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626812 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-data\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.631298 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.626909 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-credential-socket\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.631298 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.627066 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istiod-ca-cert\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.631298 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.629731 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-podinfo\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.631298 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.629993 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-envoy\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.633358 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.633331 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-token\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.633597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.633576 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljn8x\" (UniqueName: \"kubernetes.io/projected/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-kube-api-access-ljn8x\") pod \"e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.728250 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.728207 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kr8z\" (UniqueName: \"kubernetes.io/projected/015afcb4-d156-4163-a342-f84471c9dd51-kube-api-access-6kr8z\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.728924 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.728333 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-istio-data\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.728924 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.728377 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-credential-socket\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.728924 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.728416 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/015afcb4-d156-4163-a342-f84471c9dd51-istiod-ca-cert\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.728924 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.728500 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-workload-certs\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.728924 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.728639 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/015afcb4-d156-4163-a342-f84471c9dd51-istio-token\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.728924 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.728728 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-workload-socket\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.728924 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.728748 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/015afcb4-d156-4163-a342-f84471c9dd51-istio-podinfo\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.728924 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.728774 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-istio-envoy\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.730456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.730426 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-istio-data\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.731864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.731812 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-credential-socket\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.731864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.731833 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-workload-certs\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.732005 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.731884 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-istio-envoy\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.732005 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.731906 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-workload-socket\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.732078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.731999 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/015afcb4-d156-4163-a342-f84471c9dd51-istiod-ca-cert\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.732544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.732520 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/015afcb4-d156-4163-a342-f84471c9dd51-istio-podinfo\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.749390 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.749363 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kr8z\" (UniqueName: \"kubernetes.io/projected/015afcb4-d156-4163-a342-f84471c9dd51-kube-api-access-6kr8z\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.749788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.749767 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/015afcb4-d156-4163-a342-f84471c9dd51-istio-token\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.772111 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.772027 5490 generic.go:356] "Generic (PLEG): container finished" podID="459ac1eb-d4bf-4b60-badd-07fe1893f3d0" containerID="21b58edfaf6c7c047e9abe559094ad809945cd1aea40f442e2c758eb6ea9efd8" exitCode=0 Oct 31 03:46:51.772111 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.772074 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" Oct 31 03:46:51.773095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.772110 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" event={"ID":"459ac1eb-d4bf-4b60-badd-07fe1893f3d0","Type":"ContainerDied","Data":"21b58edfaf6c7c047e9abe559094ad809945cd1aea40f442e2c758eb6ea9efd8"} Oct 31 03:46:51.773095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.772171 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf" event={"ID":"459ac1eb-d4bf-4b60-badd-07fe1893f3d0","Type":"ContainerDied","Data":"5eef03c392f67ee66d7961e4ee9f1b00072643747de17bb20a2c0b6ba652929a"} Oct 31 03:46:51.773095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.772204 5490 scope.go:122] "RemoveContainer" containerID="21b58edfaf6c7c047e9abe559094ad809945cd1aea40f442e2c758eb6ea9efd8" Oct 31 03:46:51.773960 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.773842 5490 generic.go:356] "Generic (PLEG): container finished" podID="240a8cbb-b982-4490-9929-ece58bc06d5f" containerID="6f51f90454ddb1cfcb3b2ca63d7d4a468c72144e80f5b07c30f553266611e729" exitCode=0 Oct 31 03:46:51.773960 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.773894 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" event={"ID":"240a8cbb-b982-4490-9929-ece58bc06d5f","Type":"ContainerDied","Data":"6f51f90454ddb1cfcb3b2ca63d7d4a468c72144e80f5b07c30f553266611e729"} Oct 31 03:46:51.773960 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.773902 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" Oct 31 03:46:51.773960 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.773930 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5" event={"ID":"240a8cbb-b982-4490-9929-ece58bc06d5f","Type":"ContainerDied","Data":"c162e37fa7494025341a427887ab3a93e447d57984db4f1ddc99fe4736d823bb"} Oct 31 03:46:51.805111 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.804921 5490 scope.go:122] "RemoveContainer" containerID="21b58edfaf6c7c047e9abe559094ad809945cd1aea40f442e2c758eb6ea9efd8" Oct 31 03:46:51.812261 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:46:51.812228 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b58edfaf6c7c047e9abe559094ad809945cd1aea40f442e2c758eb6ea9efd8\": container with ID starting with 21b58edfaf6c7c047e9abe559094ad809945cd1aea40f442e2c758eb6ea9efd8 not found: ID does not exist" containerID="21b58edfaf6c7c047e9abe559094ad809945cd1aea40f442e2c758eb6ea9efd8" Oct 31 03:46:51.812500 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.812469 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b58edfaf6c7c047e9abe559094ad809945cd1aea40f442e2c758eb6ea9efd8"} err="failed to get container status \"21b58edfaf6c7c047e9abe559094ad809945cd1aea40f442e2c758eb6ea9efd8\": rpc error: code = NotFound desc = could not find container \"21b58edfaf6c7c047e9abe559094ad809945cd1aea40f442e2c758eb6ea9efd8\": container with ID starting with 21b58edfaf6c7c047e9abe559094ad809945cd1aea40f442e2c758eb6ea9efd8 not found: ID does not exist" Oct 31 03:46:51.812635 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.812620 5490 scope.go:122] "RemoveContainer" containerID="6f51f90454ddb1cfcb3b2ca63d7d4a468c72144e80f5b07c30f553266611e729" Oct 31 03:46:51.828767 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.813136 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:51.830279 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.819078 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:51.847972 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.847932 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf"] Oct 31 03:46:51.880131 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.880092 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-fhdrf"] Oct 31 03:46:51.882101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.882074 5490 scope.go:122] "RemoveContainer" containerID="6f51f90454ddb1cfcb3b2ca63d7d4a468c72144e80f5b07c30f553266611e729" Oct 31 03:46:51.882590 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:46:51.882566 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f51f90454ddb1cfcb3b2ca63d7d4a468c72144e80f5b07c30f553266611e729\": container with ID starting with 6f51f90454ddb1cfcb3b2ca63d7d4a468c72144e80f5b07c30f553266611e729 not found: ID does not exist" containerID="6f51f90454ddb1cfcb3b2ca63d7d4a468c72144e80f5b07c30f553266611e729" Oct 31 03:46:51.882731 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.882600 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f51f90454ddb1cfcb3b2ca63d7d4a468c72144e80f5b07c30f553266611e729"} err="failed to get container status \"6f51f90454ddb1cfcb3b2ca63d7d4a468c72144e80f5b07c30f553266611e729\": rpc error: code = NotFound desc = could not find container \"6f51f90454ddb1cfcb3b2ca63d7d4a468c72144e80f5b07c30f553266611e729\": container with ID starting with 6f51f90454ddb1cfcb3b2ca63d7d4a468c72144e80f5b07c30f553266611e729 not found: ID does not exist" Oct 31 03:46:51.901733 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.898792 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5"] Oct 31 03:46:51.901733 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.899571 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:51.907784 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:51.907757 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8385c5-openshift-default-757bf68567-zmzp5"] Oct 31 03:46:52.187983 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:46:52.187954 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015afcb4_d156_4163_a342_f84471c9dd51.slice/crio-4d6fe02dc3b811e0e4998d609b39c0b1a06080d1d7940afa95bd13f181e9d4a8 WatchSource:0}: Error finding container 4d6fe02dc3b811e0e4998d609b39c0b1a06080d1d7940afa95bd13f181e9d4a8: Status 404 returned error can't find the container with id 4d6fe02dc3b811e0e4998d609b39c0b1a06080d1d7940afa95bd13f181e9d4a8 Oct 31 03:46:52.188368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:52.188079 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn"] Oct 31 03:46:52.191927 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:52.191887 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:46:52.192032 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:52.191967 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:46:52.192032 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:52.192005 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:46:52.193496 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:52.193472 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5"] Oct 31 03:46:52.196322 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:46:52.196297 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddacc99d6_e9da_46e0_841d_14c5f54dd8fa.slice/crio-f4b582bbb3a722098ff1cbe155a8985a4fcf0a438ecfedc4a5d01b6a1be298e7 WatchSource:0}: Error finding container f4b582bbb3a722098ff1cbe155a8985a4fcf0a438ecfedc4a5d01b6a1be298e7: Status 404 returned error can't find the container with id f4b582bbb3a722098ff1cbe155a8985a4fcf0a438ecfedc4a5d01b6a1be298e7 Oct 31 03:46:52.222905 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:52.222845 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:46:52.223073 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:52.223009 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:46:52.224504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:52.223108 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:46:52.517111 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:52.517072 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:52.795781 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:52.795737 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" event={"ID":"015afcb4-d156-4163-a342-f84471c9dd51","Type":"ContainerStarted","Data":"86094c517a636ad0fb95d0395d8f04eaf0b71ce63609059fa53c252f4c2da1e7"} Oct 31 03:46:52.796010 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:52.795807 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" event={"ID":"015afcb4-d156-4163-a342-f84471c9dd51","Type":"ContainerStarted","Data":"4d6fe02dc3b811e0e4998d609b39c0b1a06080d1d7940afa95bd13f181e9d4a8"} Oct 31 03:46:52.798033 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:52.798002 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" event={"ID":"dacc99d6-e9da-46e0-841d-14c5f54dd8fa","Type":"ContainerStarted","Data":"0998ecfcad5304891c2a3145d4d7a7e940ec5d1406fc4f5311365b9172a45253"} Oct 31 03:46:52.798163 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:52.798063 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" event={"ID":"dacc99d6-e9da-46e0-841d-14c5f54dd8fa","Type":"ContainerStarted","Data":"f4b582bbb3a722098ff1cbe155a8985a4fcf0a438ecfedc4a5d01b6a1be298e7"} Oct 31 03:46:52.818104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:52.818046 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" podStartSLOduration=207.818031307 podStartE2EDuration="3m27.818031307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:46:52.815447779 +0000 UTC m=+1847.350142735" watchObservedRunningTime="2026-10-31 03:46:52.818031307 +0000 UTC m=+1847.352726261" Oct 31 03:46:52.819846 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:52.819817 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:52.819990 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:52.819879 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:52.850501 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:52.850440 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" podStartSLOduration=84.850418653 podStartE2EDuration="1m24.850418653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:46:52.848332892 +0000 UTC m=+1847.383027848" watchObservedRunningTime="2026-10-31 03:46:52.850418653 +0000 UTC m=+1847.385113610" Oct 31 03:46:53.282084 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:53.282046 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:53.579229 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:53.579148 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="240a8cbb-b982-4490-9929-ece58bc06d5f" path="/var/lib/kubelet/pods/240a8cbb-b982-4490-9929-ece58bc06d5f/volumes" Oct 31 03:46:53.579833 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:53.579807 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459ac1eb-d4bf-4b60-badd-07fe1893f3d0" path="/var/lib/kubelet/pods/459ac1eb-d4bf-4b60-badd-07fe1893f3d0/volumes" Oct 31 03:46:53.820914 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:53.820873 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 31 03:46:53.821093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:53.820924 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" podUID="dacc99d6-e9da-46e0-841d-14c5f54dd8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 31 03:46:53.825663 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:53.825638 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:53.999102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:53.997593 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:54.815653 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:54.815623 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:54.817649 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:54.817625 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:46:54.819264 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:54.819237 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 31 03:46:54.819391 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:54.819302 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" podUID="dacc99d6-e9da-46e0-841d-14c5f54dd8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 31 03:46:54.912063 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:54.912027 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:55.259235 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:55.259198 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:55.396381 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:55.396346 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:55.812018 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:55.809225 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:55.832164 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:55.832116 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 31 03:46:55.834352 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:55.834309 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" podUID="dacc99d6-e9da-46e0-841d-14c5f54dd8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 31 03:46:56.465526 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:56.465488 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:56.611000 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:56.610956 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:56.763764 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:56.759207 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:56.820001 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:56.819968 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 31 03:46:56.820155 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:56.820035 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" podUID="dacc99d6-e9da-46e0-841d-14c5f54dd8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 31 03:46:56.908447 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:56.908415 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:57.819231 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:57.819068 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 31 03:46:57.819231 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:57.819135 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" podUID="dacc99d6-e9da-46e0-841d-14c5f54dd8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 31 03:46:57.865482 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:57.851756 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:58.621845 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:58.621809 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:58.819848 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:58.819815 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Oct 31 03:46:58.820004 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:58.819873 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" podUID="dacc99d6-e9da-46e0-841d-14c5f54dd8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Oct 31 03:46:59.310560 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:59.310516 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:46:59.824896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:59.824865 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:59.825985 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:59.825280 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:46:59.825985 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:46:59.825850 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:47:00.133103 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:00.124704 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:00.790374 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:00.788375 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:01.542318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:01.542266 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:02.577935 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:02.577895 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:03.707669 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:03.707578 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:04.373243 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:04.370574 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:05.343546 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:05.343484 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:06.056853 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:06.056811 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:06.983250 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:06.983156 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:07.867216 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:07.867115 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:08.634218 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:08.634154 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:09.303024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:09.302981 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:09.729620 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:09.729583 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s"] Oct 31 03:47:09.730428 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:09.730393 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" podUID="9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9" containerName="maas-api" containerID="cri-o://394975da730c6e79753f6d5a355214ac961a6878bda14c5520ab0c651a0086d3" gracePeriod=30 Oct 31 03:47:09.804448 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:09.804407 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:09.933622 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:09.932874 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:10.137962 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.137920 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:10.215831 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.215299 5490 generic.go:356] "Generic (PLEG): container finished" podID="9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9" containerID="394975da730c6e79753f6d5a355214ac961a6878bda14c5520ab0c651a0086d3" exitCode=0 Oct 31 03:47:10.215831 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.215448 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" event={"ID":"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9","Type":"ContainerDied","Data":"394975da730c6e79753f6d5a355214ac961a6878bda14c5520ab0c651a0086d3"} Oct 31 03:47:10.215831 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.215522 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" event={"ID":"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9","Type":"ContainerDied","Data":"66a3e4e635ebc7dbd44da427a83f3a4b4b047822fef20af9be57ff37959a4efe"} Oct 31 03:47:10.215831 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.215569 5490 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66a3e4e635ebc7dbd44da427a83f3a4b4b047822fef20af9be57ff37959a4efe" Oct 31 03:47:10.273705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.272902 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" Oct 31 03:47:10.320786 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.320616 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9"] Oct 31 03:47:10.324698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.322067 5490 cpu_manager.go:414] "containerMap: removing container" podUID="9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9" containerName="maas-api" Oct 31 03:47:10.324698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.322095 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9" containerName="maas-api" Oct 31 03:47:10.324698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.322311 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9" containerName="maas-api" Oct 31 03:47:10.352701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.345651 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-kube-api-access-zlkw7\" (UniqueName: \"kubernetes.io/projected/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-kube-api-access-zlkw7\") pod \"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9\" (UID: \"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9\") " Oct 31 03:47:10.352701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.345766 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-maas-api-tls\") pod \"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9\" (UID: \"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9\") " Oct 31 03:47:10.352701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.345847 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-openshift-service-ca\") pod \"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9\" (UID: \"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9\") " Oct 31 03:47:10.352701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.346075 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-metrics-certs\") pod \"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9\" (UID: \"9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9\") " Oct 31 03:47:10.352701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.352465 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-openshift-service-ca" pod "9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9" (UID: "9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:10.360699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.355078 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:10.364703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.362959 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.388697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.383271 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-880e04-openshift-default-dockercfg-8z2d9\"" Oct 31 03:47:10.404706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.400003 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-metrics-certs" pod "9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9" (UID: "9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:10.404706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.403051 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-kube-api-access-zlkw7" pod "9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9" (UID: "9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9"). InnerVolumeSpecName "kube-api-access-zlkw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:10.404706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.403562 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-maas-api-tls" pod "9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9" (UID: "9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:10.412706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.405765 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9"] Oct 31 03:47:10.457721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.456713 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-istio-data\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.457721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.456775 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-istio-envoy\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.457721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.456824 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3a29402-5f18-4922-b80d-91be39273cd5-istio-podinfo\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.457721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.457049 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3a29402-5f18-4922-b80d-91be39273cd5-istio-token\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.457721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.457156 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3a29402-5f18-4922-b80d-91be39273cd5-istiod-ca-cert\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.457721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.457222 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-workload-socket\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.457721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.457349 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-workload-certs\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.457721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.457434 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-credential-socket\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.457721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.457470 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mvjt\" (UniqueName: \"kubernetes.io/projected/b3a29402-5f18-4922-b80d-91be39273cd5-kube-api-access-4mvjt\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.457721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.457537 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:10.457721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.457550 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlkw7\" (UniqueName: \"kubernetes.io/projected/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-kube-api-access-zlkw7\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:10.457721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.457562 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:10.559646 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.559608 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3a29402-5f18-4922-b80d-91be39273cd5-istio-token\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.559865 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.559740 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3a29402-5f18-4922-b80d-91be39273cd5-istiod-ca-cert\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.559865 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.559791 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-workload-socket\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.560003 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.559918 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-workload-certs\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.560003 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.559993 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-credential-socket\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.560114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.560028 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mvjt\" (UniqueName: \"kubernetes.io/projected/b3a29402-5f18-4922-b80d-91be39273cd5-kube-api-access-4mvjt\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.560114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.560048 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-istio-data\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.560114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.560071 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-istio-envoy\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.560114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.560101 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3a29402-5f18-4922-b80d-91be39273cd5-istio-podinfo\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.561368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.561339 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:10.562027 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.561919 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-workload-certs\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.562967 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.562940 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3a29402-5f18-4922-b80d-91be39273cd5-istiod-ca-cert\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.563210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.563190 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3a29402-5f18-4922-b80d-91be39273cd5-istio-podinfo\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.563816 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.563789 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-workload-socket\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.565128 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.564876 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-credential-socket\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.565128 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.564912 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-istio-data\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.567268 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.567245 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-istio-envoy\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.573150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.573088 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3a29402-5f18-4922-b80d-91be39273cd5-istio-token\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.573150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.573123 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mvjt\" (UniqueName: \"kubernetes.io/projected/b3a29402-5f18-4922-b80d-91be39273cd5-kube-api-access-4mvjt\") pod \"e2e-inf-b-880e04-openshift-default-f55d69847-g77p9\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:10.708507 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.708470 5490 scope.go:122] "RemoveContainer" containerID="394975da730c6e79753f6d5a355214ac961a6878bda14c5520ab0c651a0086d3" Oct 31 03:47:10.839873 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:10.839790 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:11.136838 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:11.136803 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9"] Oct 31 03:47:11.138155 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:47:11.138127 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a29402_5f18_4922_b80d_91be39273cd5.slice/crio-1ca06930a65996802f3a3a80c57fd78d0738f8a4f2305f92bcf9429982a10f84 WatchSource:0}: Error finding container 1ca06930a65996802f3a3a80c57fd78d0738f8a4f2305f92bcf9429982a10f84: Status 404 returned error can't find the container with id 1ca06930a65996802f3a3a80c57fd78d0738f8a4f2305f92bcf9429982a10f84 Oct 31 03:47:11.141426 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:11.141393 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:47:11.141544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:11.141474 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:47:11.141544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:11.141506 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:47:11.225618 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:11.225582 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" event={"ID":"b3a29402-5f18-4922-b80d-91be39273cd5","Type":"ContainerStarted","Data":"1ca06930a65996802f3a3a80c57fd78d0738f8a4f2305f92bcf9429982a10f84"} Oct 31 03:47:11.225618 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:11.225609 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" Oct 31 03:47:11.270114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:11.269591 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:11.304269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:11.304225 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s"] Oct 31 03:47:11.311385 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:11.311355 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s"] Oct 31 03:47:11.579458 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:11.579421 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9" path="/var/lib/kubelet/pods/9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9/volumes" Oct 31 03:47:12.045330 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:12.044327 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:12.265823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:12.265778 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" event={"ID":"b3a29402-5f18-4922-b80d-91be39273cd5","Type":"ContainerStarted","Data":"a4fced6b2f7e94fddef9d6ee41b6cde856d2684f6a604986b7c62978b73d06ca"} Oct 31 03:47:12.295476 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:12.295376 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" podStartSLOduration=2.295362338 podStartE2EDuration="2.295362338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:12.289974553 +0000 UTC m=+1866.824669509" watchObservedRunningTime="2026-10-31 03:47:12.295362338 +0000 UTC m=+1866.830057293" Oct 31 03:47:12.797061 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:12.797012 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:12.860712 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:12.860291 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:12.878697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:12.874803 5490 patch_prober.go:29] interesting pod/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:47:12.878697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:12.874868 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" podUID="b3a29402-5f18-4922-b80d-91be39273cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 03:47:12.959964 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:12.959897 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6f28b4-68c48c8f5f-d9w2s" podUID="9e3cbb24-ff7d-4913-ad1e-f7bd39470ce9" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.115:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:47:13.662761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:13.662655 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:13.849155 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:13.849114 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:13.850093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:13.850066 5490 patch_prober.go:29] interesting pod/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:47:13.850360 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:13.850312 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" podUID="b3a29402-5f18-4922-b80d-91be39273cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 03:47:14.145671 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:14.145398 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:14.852702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:14.852421 5490 patch_prober.go:29] interesting pod/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:47:14.852702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:14.852486 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" podUID="b3a29402-5f18-4922-b80d-91be39273cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 03:47:15.272702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:15.271326 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:15.854238 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:15.852785 5490 patch_prober.go:29] interesting pod/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:47:15.854238 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:15.852856 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" podUID="b3a29402-5f18-4922-b80d-91be39273cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 03:47:15.945698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:15.944772 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:16.836711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:16.832256 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:16.852702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:16.849163 5490 patch_prober.go:29] interesting pod/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:47:16.852702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:16.849232 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" podUID="b3a29402-5f18-4922-b80d-91be39273cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 03:47:17.158551 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:47:17.158511 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:47:17.552767 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:17.552291 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:17.681360 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:17.681331 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:17.815753 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:17.812192 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q"] Oct 31 03:47:17.815753 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:17.813192 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" podUID="82755388-258d-4de0-bdfe-59073fb3dfac" containerName="istio-proxy" containerID="cri-o://966af05b50546a53479e02ae0dfcda86bc45800ba2292905b9052c8064c8673f" gracePeriod=30 Oct 31 03:47:17.825832 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:17.823251 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:17.846740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:17.843261 5490 patch_prober.go:29] interesting pod/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 31 03:47:17.846740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:17.843326 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" podUID="b3a29402-5f18-4922-b80d-91be39273cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 31 03:47:18.010701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:18.000383 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:18.752698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:18.749793 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:18.934786 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:18.930613 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:18.934786 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:18.931279 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:18.934786 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:18.932830 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:47:19.139983 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:19.139890 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:19.277021 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:19.276980 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:19.423698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:19.418171 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:20.121707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:20.121600 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:20.777356 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:20.777313 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:21.243113 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:21.243076 5490 patch_prober.go:29] interesting pod/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:47:21.243532 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:21.243137 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" podUID="82755388-258d-4de0-bdfe-59073fb3dfac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:47:21.488806 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:21.488764 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:22.192886 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:22.192852 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:22.884830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:22.884796 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:23.188094 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.188064 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:47:23.223651 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.223614 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss"] Oct 31 03:47:23.225150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.225120 5490 cpu_manager.go:414] "containerMap: removing container" podUID="82755388-258d-4de0-bdfe-59073fb3dfac" containerName="istio-proxy" Oct 31 03:47:23.225150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.225151 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82755388-258d-4de0-bdfe-59073fb3dfac" containerName="istio-proxy" Oct 31 03:47:23.225473 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.225452 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="82755388-258d-4de0-bdfe-59073fb3dfac" containerName="istio-proxy" Oct 31 03:47:23.238757 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.238723 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" Oct 31 03:47:23.244212 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.244128 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss"] Oct 31 03:47:23.251127 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.250981 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b81d65-serving-cert\"" Oct 31 03:47:23.364578 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.364547 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-istio-envoy\") pod \"82755388-258d-4de0-bdfe-59073fb3dfac\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " Oct 31 03:47:23.364798 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.364601 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-workload-socket\") pod \"82755388-258d-4de0-bdfe-59073fb3dfac\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " Oct 31 03:47:23.364798 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.364671 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-credential-socket\") pod \"82755388-258d-4de0-bdfe-59073fb3dfac\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " Oct 31 03:47:23.364798 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.364771 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82755388-258d-4de0-bdfe-59073fb3dfac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82755388-258d-4de0-bdfe-59073fb3dfac-istiod-ca-cert\") pod \"82755388-258d-4de0-bdfe-59073fb3dfac\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " Oct 31 03:47:23.364988 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.364822 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82755388-258d-4de0-bdfe-59073fb3dfac-kube-api-access-6sc8p\" (UniqueName: \"kubernetes.io/projected/82755388-258d-4de0-bdfe-59073fb3dfac-kube-api-access-6sc8p\") pod \"82755388-258d-4de0-bdfe-59073fb3dfac\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " Oct 31 03:47:23.364988 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.364956 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-workload-socket" pod "82755388-258d-4de0-bdfe-59073fb3dfac" (UID: "82755388-258d-4de0-bdfe-59073fb3dfac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:23.364988 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.364965 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82755388-258d-4de0-bdfe-59073fb3dfac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82755388-258d-4de0-bdfe-59073fb3dfac-istio-podinfo\") pod \"82755388-258d-4de0-bdfe-59073fb3dfac\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " Oct 31 03:47:23.365896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.365024 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-istio-data\") pod \"82755388-258d-4de0-bdfe-59073fb3dfac\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " Oct 31 03:47:23.365896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.365103 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82755388-258d-4de0-bdfe-59073fb3dfac-istio-token\" (UniqueName: \"kubernetes.io/projected/82755388-258d-4de0-bdfe-59073fb3dfac-istio-token\") pod \"82755388-258d-4de0-bdfe-59073fb3dfac\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " Oct 31 03:47:23.365896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.365235 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-workload-certs\") pod \"82755388-258d-4de0-bdfe-59073fb3dfac\" (UID: \"82755388-258d-4de0-bdfe-59073fb3dfac\") " Oct 31 03:47:23.365896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.365844 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da913fbf-0ba8-4607-be4c-5197a22d1a7d-metrics-certs\") pod \"maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss\" (UID: \"da913fbf-0ba8-4607-be4c-5197a22d1a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" Oct 31 03:47:23.367195 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.366057 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-credential-socket" pod "82755388-258d-4de0-bdfe-59073fb3dfac" (UID: "82755388-258d-4de0-bdfe-59073fb3dfac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:23.367195 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.366071 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da913fbf-0ba8-4607-be4c-5197a22d1a7d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss\" (UID: \"da913fbf-0ba8-4607-be4c-5197a22d1a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" Oct 31 03:47:23.367195 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.366322 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da913fbf-0ba8-4607-be4c-5197a22d1a7d-maas-api-tls\") pod \"maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss\" (UID: \"da913fbf-0ba8-4607-be4c-5197a22d1a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" Oct 31 03:47:23.367195 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.366394 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgl7j\" (UniqueName: \"kubernetes.io/projected/da913fbf-0ba8-4607-be4c-5197a22d1a7d-kube-api-access-hgl7j\") pod \"maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss\" (UID: \"da913fbf-0ba8-4607-be4c-5197a22d1a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" Oct 31 03:47:23.367195 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.366580 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:23.367195 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.366598 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:23.367969 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.367656 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82755388-258d-4de0-bdfe-59073fb3dfac-istiod-ca-cert" pod "82755388-258d-4de0-bdfe-59073fb3dfac" (UID: "82755388-258d-4de0-bdfe-59073fb3dfac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:23.367969 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.367833 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-istio-data" pod "82755388-258d-4de0-bdfe-59073fb3dfac" (UID: "82755388-258d-4de0-bdfe-59073fb3dfac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:23.368512 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.368334 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-workload-certs" pod "82755388-258d-4de0-bdfe-59073fb3dfac" (UID: "82755388-258d-4de0-bdfe-59073fb3dfac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:23.376464 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.376435 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82755388-258d-4de0-bdfe-59073fb3dfac-kube-api-access-6sc8p" pod "82755388-258d-4de0-bdfe-59073fb3dfac" (UID: "82755388-258d-4de0-bdfe-59073fb3dfac"). InnerVolumeSpecName "kube-api-access-6sc8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:23.376565 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.376461 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82755388-258d-4de0-bdfe-59073fb3dfac-istio-token" pod "82755388-258d-4de0-bdfe-59073fb3dfac" (UID: "82755388-258d-4de0-bdfe-59073fb3dfac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:23.376565 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.376488 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82755388-258d-4de0-bdfe-59073fb3dfac-istio-podinfo" pod "82755388-258d-4de0-bdfe-59073fb3dfac" (UID: "82755388-258d-4de0-bdfe-59073fb3dfac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:47:23.376565 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.376495 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-istio-envoy" pod "82755388-258d-4de0-bdfe-59073fb3dfac" (UID: "82755388-258d-4de0-bdfe-59073fb3dfac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:23.468532 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.468481 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da913fbf-0ba8-4607-be4c-5197a22d1a7d-metrics-certs\") pod \"maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss\" (UID: \"da913fbf-0ba8-4607-be4c-5197a22d1a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" Oct 31 03:47:23.468731 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.468644 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da913fbf-0ba8-4607-be4c-5197a22d1a7d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss\" (UID: \"da913fbf-0ba8-4607-be4c-5197a22d1a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" Oct 31 03:47:23.468843 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.468827 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da913fbf-0ba8-4607-be4c-5197a22d1a7d-maas-api-tls\") pod \"maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss\" (UID: \"da913fbf-0ba8-4607-be4c-5197a22d1a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" Oct 31 03:47:23.468905 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.468890 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgl7j\" (UniqueName: \"kubernetes.io/projected/da913fbf-0ba8-4607-be4c-5197a22d1a7d-kube-api-access-hgl7j\") pod \"maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss\" (UID: \"da913fbf-0ba8-4607-be4c-5197a22d1a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" Oct 31 03:47:23.469021 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.469005 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82755388-258d-4de0-bdfe-59073fb3dfac-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:23.469089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.469027 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sc8p\" (UniqueName: \"kubernetes.io/projected/82755388-258d-4de0-bdfe-59073fb3dfac-kube-api-access-6sc8p\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:23.469089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.469043 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82755388-258d-4de0-bdfe-59073fb3dfac-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:23.469089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.469058 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:23.469089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.469073 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82755388-258d-4de0-bdfe-59073fb3dfac-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:23.469089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.469088 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:23.469375 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.469102 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82755388-258d-4de0-bdfe-59073fb3dfac-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:23.469486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.469461 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da913fbf-0ba8-4607-be4c-5197a22d1a7d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss\" (UID: \"da913fbf-0ba8-4607-be4c-5197a22d1a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" Oct 31 03:47:23.472092 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.472064 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da913fbf-0ba8-4607-be4c-5197a22d1a7d-maas-api-tls\") pod \"maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss\" (UID: \"da913fbf-0ba8-4607-be4c-5197a22d1a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" Oct 31 03:47:23.472252 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.472118 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da913fbf-0ba8-4607-be4c-5197a22d1a7d-metrics-certs\") pod \"maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss\" (UID: \"da913fbf-0ba8-4607-be4c-5197a22d1a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" Oct 31 03:47:23.479565 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.479541 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgl7j\" (UniqueName: \"kubernetes.io/projected/da913fbf-0ba8-4607-be4c-5197a22d1a7d-kube-api-access-hgl7j\") pod \"maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss\" (UID: \"da913fbf-0ba8-4607-be4c-5197a22d1a7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" Oct 31 03:47:23.557892 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.557850 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" Oct 31 03:47:23.681520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.681480 5490 generic.go:356] "Generic (PLEG): container finished" podID="82755388-258d-4de0-bdfe-59073fb3dfac" containerID="966af05b50546a53479e02ae0dfcda86bc45800ba2292905b9052c8064c8673f" exitCode=0 Oct 31 03:47:23.681727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.681557 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" event={"ID":"82755388-258d-4de0-bdfe-59073fb3dfac","Type":"ContainerDied","Data":"966af05b50546a53479e02ae0dfcda86bc45800ba2292905b9052c8064c8673f"} Oct 31 03:47:23.681727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.681571 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" Oct 31 03:47:23.681727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.681618 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q" event={"ID":"82755388-258d-4de0-bdfe-59073fb3dfac","Type":"ContainerDied","Data":"faeb723318aa4ea89e9d5dc763d371a6a583b7df429e2aa11a9f7867f8fa6688"} Oct 31 03:47:23.681727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.681652 5490 scope.go:122] "RemoveContainer" containerID="966af05b50546a53479e02ae0dfcda86bc45800ba2292905b9052c8064c8673f" Oct 31 03:47:23.711670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.711638 5490 scope.go:122] "RemoveContainer" containerID="c561dcbe0b0d736ebfdbe624c225a88e00c975cb9d15a6e0d036cea2cfa0a9fa" Oct 31 03:47:23.781503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.781430 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q"] Oct 31 03:47:23.789272 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.789206 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6f28b4-openshift-default-f9559fcb5-pqj5q"] Oct 31 03:47:23.804302 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.804280 5490 scope.go:122] "RemoveContainer" containerID="966af05b50546a53479e02ae0dfcda86bc45800ba2292905b9052c8064c8673f" Oct 31 03:47:23.804792 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:47:23.804758 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"966af05b50546a53479e02ae0dfcda86bc45800ba2292905b9052c8064c8673f\": container with ID starting with 966af05b50546a53479e02ae0dfcda86bc45800ba2292905b9052c8064c8673f not found: ID does not exist" containerID="966af05b50546a53479e02ae0dfcda86bc45800ba2292905b9052c8064c8673f" Oct 31 03:47:23.807829 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.807796 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966af05b50546a53479e02ae0dfcda86bc45800ba2292905b9052c8064c8673f"} err="failed to get container status \"966af05b50546a53479e02ae0dfcda86bc45800ba2292905b9052c8064c8673f\": rpc error: code = NotFound desc = could not find container \"966af05b50546a53479e02ae0dfcda86bc45800ba2292905b9052c8064c8673f\": container with ID starting with 966af05b50546a53479e02ae0dfcda86bc45800ba2292905b9052c8064c8673f not found: ID does not exist" Oct 31 03:47:23.807971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.807956 5490 scope.go:122] "RemoveContainer" containerID="c561dcbe0b0d736ebfdbe624c225a88e00c975cb9d15a6e0d036cea2cfa0a9fa" Oct 31 03:47:23.808526 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:47:23.808500 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c561dcbe0b0d736ebfdbe624c225a88e00c975cb9d15a6e0d036cea2cfa0a9fa\": container with ID starting with c561dcbe0b0d736ebfdbe624c225a88e00c975cb9d15a6e0d036cea2cfa0a9fa not found: ID does not exist" containerID="c561dcbe0b0d736ebfdbe624c225a88e00c975cb9d15a6e0d036cea2cfa0a9fa" Oct 31 03:47:23.808624 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.808531 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c561dcbe0b0d736ebfdbe624c225a88e00c975cb9d15a6e0d036cea2cfa0a9fa"} err="failed to get container status \"c561dcbe0b0d736ebfdbe624c225a88e00c975cb9d15a6e0d036cea2cfa0a9fa\": rpc error: code = NotFound desc = could not find container \"c561dcbe0b0d736ebfdbe624c225a88e00c975cb9d15a6e0d036cea2cfa0a9fa\": container with ID starting with c561dcbe0b0d736ebfdbe624c225a88e00c975cb9d15a6e0d036cea2cfa0a9fa not found: ID does not exist" Oct 31 03:47:23.863886 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.863851 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:23.887216 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:23.887186 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss"] Oct 31 03:47:24.693441 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:24.693398 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" event={"ID":"da913fbf-0ba8-4607-be4c-5197a22d1a7d","Type":"ContainerStarted","Data":"ab9172a4dc9016f8dba10b957efc70df500d6f2605c27e6dd2ff7a99262b480a"} Oct 31 03:47:24.693672 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:24.693473 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" Oct 31 03:47:24.693672 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:24.693496 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" event={"ID":"da913fbf-0ba8-4607-be4c-5197a22d1a7d","Type":"ContainerStarted","Data":"a425344e776d6d050265787d2d612ae815783a66b0b835bf358210c0ac71965b"} Oct 31 03:47:24.717050 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:24.716996 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" podStartSLOduration=7.054915274 podStartE2EDuration="7.716983177s" totalImagesPullingTime="662.067903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:24.715935828 +0000 UTC m=+1879.250630784" watchObservedRunningTime="2026-10-31 03:47:24.716983177 +0000 UTC m=+1879.251678132" Oct 31 03:47:24.797533 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:24.797491 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:25.581601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:25.581561 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82755388-258d-4de0-bdfe-59073fb3dfac" path="/var/lib/kubelet/pods/82755388-258d-4de0-bdfe-59073fb3dfac/volumes" Oct 31 03:47:25.875114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:25.874642 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:26.558092 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:26.558052 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:26.731506 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:26.712204 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:27.878738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:27.877004 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:28.035700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:28.035306 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:28.404164 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:28.404127 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:28.670443 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:28.670349 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:29.645333 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:29.643789 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:30.697622 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:30.697522 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:30.739962 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:30.739925 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" Oct 31 03:47:31.511746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:31.506872 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:32.157023 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:32.152480 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:32.784979 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:32.784941 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:33.671035 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:33.670990 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:34.481445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:34.481404 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:35.140754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:35.136997 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:35.828102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:35.827248 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:36.594305 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:36.594266 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:37.286466 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:37.286430 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:37.966425 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:37.966389 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:38.980549 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:38.980507 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:39.144444 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:39.144382 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:39.292036 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:39.291945 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:40.108401 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:40.108361 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:40.799273 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:40.799231 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:41.600706 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:41.600653 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:42.685122 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:42.674532 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:43.453986 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:43.452622 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:44.056868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:44.056790 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:44.727148 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:44.727114 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:45.481554 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:45.481517 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:46.267429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:46.267391 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:46.425410 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:46.425371 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:46.760701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:46.760633 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:47.179398 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:47:47.179356 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:47:47.294701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:47.282961 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:47.810625 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:47.810183 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:47.987595 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:47.987554 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:48.747819 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:48.747776 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:48.894248 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:48.894204 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:49.068757 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:49.060618 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:50.187660 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:50.187623 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:50.329903 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:50.329860 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:50.471266 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:50.471176 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:50.995964 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:50.995934 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs"] Oct 31 03:47:50.996465 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:50.996430 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" podUID="2cf3d600-6bdb-46fe-857b-219e0c3ba93c" containerName="maas-api" containerID="cri-o://54b4f4bb363ff26938fbc292eec3023772a87e303ab6884494598dce69f0ac81" gracePeriod=30 Oct 31 03:47:51.088182 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.087128 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:51.677440 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.677411 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:51.802231 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.802105 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:51.899909 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.899856 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" Oct 31 03:47:51.961744 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.961601 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4"] Oct 31 03:47:51.963719 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.963645 5490 cpu_manager.go:414] "containerMap: removing container" podUID="2cf3d600-6bdb-46fe-857b-219e0c3ba93c" containerName="maas-api" Oct 31 03:47:51.963719 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.963693 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf3d600-6bdb-46fe-857b-219e0c3ba93c" containerName="maas-api" Oct 31 03:47:51.963913 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.963732 5490 cpu_manager.go:414] "containerMap: removing container" podUID="82755388-258d-4de0-bdfe-59073fb3dfac" containerName="istio-proxy" Oct 31 03:47:51.963913 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.963740 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82755388-258d-4de0-bdfe-59073fb3dfac" containerName="istio-proxy" Oct 31 03:47:51.964018 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.963999 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf3d600-6bdb-46fe-857b-219e0c3ba93c" containerName="maas-api" Oct 31 03:47:51.964065 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.964027 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="82755388-258d-4de0-bdfe-59073fb3dfac" containerName="istio-proxy" Oct 31 03:47:51.972872 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.972843 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" Oct 31 03:47:51.978114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.978086 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-b8fd22-serving-cert\"" Oct 31 03:47:51.989577 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.985299 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4"] Oct 31 03:47:51.994832 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.994798 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-maas-api-tls\") pod \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\" (UID: \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\") " Oct 31 03:47:51.995041 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.995019 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-kube-api-access-t875r\" (UniqueName: \"kubernetes.io/projected/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-kube-api-access-t875r\") pod \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\" (UID: \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\") " Oct 31 03:47:51.995120 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.995080 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-metrics-certs\") pod \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\" (UID: \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\") " Oct 31 03:47:51.995379 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.995358 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-openshift-service-ca\") pod \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\" (UID: \"2cf3d600-6bdb-46fe-857b-219e0c3ba93c\") " Oct 31 03:47:51.995938 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.995908 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h42jp\" (UniqueName: \"kubernetes.io/projected/b7c0d839-b3a4-463b-a3a8-3733609d4b89-kube-api-access-h42jp\") pod \"maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4\" (UID: \"b7c0d839-b3a4-463b-a3a8-3733609d4b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" Oct 31 03:47:51.996320 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.996297 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c0d839-b3a4-463b-a3a8-3733609d4b89-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4\" (UID: \"b7c0d839-b3a4-463b-a3a8-3733609d4b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" Oct 31 03:47:51.996911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.996336 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c0d839-b3a4-463b-a3a8-3733609d4b89-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4\" (UID: \"b7c0d839-b3a4-463b-a3a8-3733609d4b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" Oct 31 03:47:51.996911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.996381 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7c0d839-b3a4-463b-a3a8-3733609d4b89-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4\" (UID: \"b7c0d839-b3a4-463b-a3a8-3733609d4b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" Oct 31 03:47:51.997236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:51.997217 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-openshift-service-ca" pod "2cf3d600-6bdb-46fe-857b-219e0c3ba93c" (UID: "2cf3d600-6bdb-46fe-857b-219e0c3ba93c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:52.000813 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.000784 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-maas-api-tls" pod "2cf3d600-6bdb-46fe-857b-219e0c3ba93c" (UID: "2cf3d600-6bdb-46fe-857b-219e0c3ba93c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:52.000961 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.000878 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-kube-api-access-t875r" pod "2cf3d600-6bdb-46fe-857b-219e0c3ba93c" (UID: "2cf3d600-6bdb-46fe-857b-219e0c3ba93c"). InnerVolumeSpecName "kube-api-access-t875r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:52.001491 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.001468 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-metrics-certs" pod "2cf3d600-6bdb-46fe-857b-219e0c3ba93c" (UID: "2cf3d600-6bdb-46fe-857b-219e0c3ba93c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:47:52.020495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.020455 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:52.115700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.113329 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h42jp\" (UniqueName: \"kubernetes.io/projected/b7c0d839-b3a4-463b-a3a8-3733609d4b89-kube-api-access-h42jp\") pod \"maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4\" (UID: \"b7c0d839-b3a4-463b-a3a8-3733609d4b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" Oct 31 03:47:52.115700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.113670 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c0d839-b3a4-463b-a3a8-3733609d4b89-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4\" (UID: \"b7c0d839-b3a4-463b-a3a8-3733609d4b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" Oct 31 03:47:52.115700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.113734 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c0d839-b3a4-463b-a3a8-3733609d4b89-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4\" (UID: \"b7c0d839-b3a4-463b-a3a8-3733609d4b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" Oct 31 03:47:52.115700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.113778 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7c0d839-b3a4-463b-a3a8-3733609d4b89-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4\" (UID: \"b7c0d839-b3a4-463b-a3a8-3733609d4b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" Oct 31 03:47:52.115700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.113998 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:52.115700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.114016 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:52.115700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.114032 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t875r\" (UniqueName: \"kubernetes.io/projected/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-kube-api-access-t875r\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:52.115700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.114046 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cf3d600-6bdb-46fe-857b-219e0c3ba93c-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:52.121165 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.116886 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c0d839-b3a4-463b-a3a8-3733609d4b89-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4\" (UID: \"b7c0d839-b3a4-463b-a3a8-3733609d4b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" Oct 31 03:47:52.145703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.144639 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c0d839-b3a4-463b-a3a8-3733609d4b89-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4\" (UID: \"b7c0d839-b3a4-463b-a3a8-3733609d4b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" Oct 31 03:47:52.171707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.171328 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:52.194700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.172265 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7c0d839-b3a4-463b-a3a8-3733609d4b89-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4\" (UID: \"b7c0d839-b3a4-463b-a3a8-3733609d4b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" Oct 31 03:47:52.194700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.189973 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h42jp\" (UniqueName: \"kubernetes.io/projected/b7c0d839-b3a4-463b-a3a8-3733609d4b89-kube-api-access-h42jp\") pod \"maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4\" (UID: \"b7c0d839-b3a4-463b-a3a8-3733609d4b89\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" Oct 31 03:47:52.307756 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.305416 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" Oct 31 03:47:52.350778 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.339666 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:52.493711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.491375 5490 generic.go:356] "Generic (PLEG): container finished" podID="2cf3d600-6bdb-46fe-857b-219e0c3ba93c" containerID="54b4f4bb363ff26938fbc292eec3023772a87e303ab6884494598dce69f0ac81" exitCode=0 Oct 31 03:47:52.493711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.491533 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" event={"ID":"2cf3d600-6bdb-46fe-857b-219e0c3ba93c","Type":"ContainerDied","Data":"54b4f4bb363ff26938fbc292eec3023772a87e303ab6884494598dce69f0ac81"} Oct 31 03:47:52.493711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.491538 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" Oct 31 03:47:52.493711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.491570 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs" event={"ID":"2cf3d600-6bdb-46fe-857b-219e0c3ba93c","Type":"ContainerDied","Data":"0b18eaabaaf99a2c3f17970cc9230578b9681d0942f6677e241a1ba14d87661a"} Oct 31 03:47:52.493711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.491596 5490 scope.go:122] "RemoveContainer" containerID="54b4f4bb363ff26938fbc292eec3023772a87e303ab6884494598dce69f0ac81" Oct 31 03:47:52.538987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.538951 5490 scope.go:122] "RemoveContainer" containerID="54b4f4bb363ff26938fbc292eec3023772a87e303ab6884494598dce69f0ac81" Oct 31 03:47:52.540299 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:47:52.540125 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b4f4bb363ff26938fbc292eec3023772a87e303ab6884494598dce69f0ac81\": container with ID starting with 54b4f4bb363ff26938fbc292eec3023772a87e303ab6884494598dce69f0ac81 not found: ID does not exist" containerID="54b4f4bb363ff26938fbc292eec3023772a87e303ab6884494598dce69f0ac81" Oct 31 03:47:52.540299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.540203 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b4f4bb363ff26938fbc292eec3023772a87e303ab6884494598dce69f0ac81"} err="failed to get container status \"54b4f4bb363ff26938fbc292eec3023772a87e303ab6884494598dce69f0ac81\": rpc error: code = NotFound desc = could not find container \"54b4f4bb363ff26938fbc292eec3023772a87e303ab6884494598dce69f0ac81\": container with ID starting with 54b4f4bb363ff26938fbc292eec3023772a87e303ab6884494598dce69f0ac81 not found: ID does not exist" Oct 31 03:47:52.546556 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.546519 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb"] Oct 31 03:47:52.547071 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.546907 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" podUID="93fab1f3-4ba5-46fa-a9f6-001946891f5b" containerName="istio-proxy" containerID="cri-o://80ceb5748003ba6aad148b3cbff454e941c905cce04047854876fbda303e7923" gracePeriod=30 Oct 31 03:47:52.554857 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.554830 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs"] Oct 31 03:47:52.564970 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.564884 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b2dc82-587769dc7f-7cmjs"] Oct 31 03:47:52.846084 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:47:52.846036 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c0d839_b3a4_463b_a3a8_3733609d4b89.slice/crio-24ab5e0e007b254608cdc62961f270028130d45b567c169816821b3d05fe6554 WatchSource:0}: Error finding container 24ab5e0e007b254608cdc62961f270028130d45b567c169816821b3d05fe6554: Status 404 returned error can't find the container with id 24ab5e0e007b254608cdc62961f270028130d45b567c169816821b3d05fe6554 Oct 31 03:47:52.849231 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:52.849197 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4"] Oct 31 03:47:53.077552 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:53.077508 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:53.374710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:53.374659 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:53.510408 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:53.510371 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" event={"ID":"b7c0d839-b3a4-463b-a3a8-3733609d4b89","Type":"ContainerStarted","Data":"4806f5c5bfc3514bb6e3f351df9f6f8c5248b8f144eb7263ac29a806d51a3ed2"} Oct 31 03:47:53.510638 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:53.510443 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" event={"ID":"b7c0d839-b3a4-463b-a3a8-3733609d4b89","Type":"ContainerStarted","Data":"24ab5e0e007b254608cdc62961f270028130d45b567c169816821b3d05fe6554"} Oct 31 03:47:53.510638 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:53.510492 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" Oct 31 03:47:53.514458 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:53.514026 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:53.566091 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:53.562731 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" podStartSLOduration=3.113061748 podStartE2EDuration="3.562707287s" totalImagesPullingTime="449.645539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:53.536229377 +0000 UTC m=+1908.070924362" watchObservedRunningTime="2026-10-31 03:47:53.562707287 +0000 UTC m=+1908.097402246" Oct 31 03:47:53.674697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:53.674536 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf3d600-6bdb-46fe-857b-219e0c3ba93c" path="/var/lib/kubelet/pods/2cf3d600-6bdb-46fe-857b-219e0c3ba93c/volumes" Oct 31 03:47:54.313169 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:54.313127 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:55.100294 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:55.100253 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:55.773715 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:55.772077 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:56.499302 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:56.499266 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:57.193518 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:57.187832 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:57.828705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:57.824843 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:57.984144 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:57.984111 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:47:58.144542 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.144502 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-token\" (UniqueName: \"kubernetes.io/projected/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-token\") pod \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " Oct 31 03:47:58.145475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.144736 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istiod-ca-cert\") pod \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " Oct 31 03:47:58.145475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.144784 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93fab1f3-4ba5-46fa-a9f6-001946891f5b-kube-api-access-srxjn\" (UniqueName: \"kubernetes.io/projected/93fab1f3-4ba5-46fa-a9f6-001946891f5b-kube-api-access-srxjn\") pod \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " Oct 31 03:47:58.145475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.144886 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-workload-certs\") pod \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " Oct 31 03:47:58.145475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.145005 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-credential-socket\") pod \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " Oct 31 03:47:58.145475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.145116 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-workload-socket\") pod \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " Oct 31 03:47:58.145475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.145172 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-data\") pod \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " Oct 31 03:47:58.145475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.145244 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-envoy\") pod \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " Oct 31 03:47:58.145475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.145309 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-podinfo\") pod \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\" (UID: \"93fab1f3-4ba5-46fa-a9f6-001946891f5b\") " Oct 31 03:47:58.147024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.145864 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istiod-ca-cert" pod "93fab1f3-4ba5-46fa-a9f6-001946891f5b" (UID: "93fab1f3-4ba5-46fa-a9f6-001946891f5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:47:58.147024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.146108 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:58.147024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.146393 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-credential-socket" pod "93fab1f3-4ba5-46fa-a9f6-001946891f5b" (UID: "93fab1f3-4ba5-46fa-a9f6-001946891f5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:58.147024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.146473 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-workload-socket" pod "93fab1f3-4ba5-46fa-a9f6-001946891f5b" (UID: "93fab1f3-4ba5-46fa-a9f6-001946891f5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:58.147024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.146544 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-data" pod "93fab1f3-4ba5-46fa-a9f6-001946891f5b" (UID: "93fab1f3-4ba5-46fa-a9f6-001946891f5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:58.147024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.146558 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-workload-certs" pod "93fab1f3-4ba5-46fa-a9f6-001946891f5b" (UID: "93fab1f3-4ba5-46fa-a9f6-001946891f5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:58.148882 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.147122 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-token" pod "93fab1f3-4ba5-46fa-a9f6-001946891f5b" (UID: "93fab1f3-4ba5-46fa-a9f6-001946891f5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:58.148882 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.148778 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-envoy" pod "93fab1f3-4ba5-46fa-a9f6-001946891f5b" (UID: "93fab1f3-4ba5-46fa-a9f6-001946891f5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:47:58.149022 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.149002 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93fab1f3-4ba5-46fa-a9f6-001946891f5b-kube-api-access-srxjn" pod "93fab1f3-4ba5-46fa-a9f6-001946891f5b" (UID: "93fab1f3-4ba5-46fa-a9f6-001946891f5b"). InnerVolumeSpecName "kube-api-access-srxjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:47:58.149022 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.149010 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-podinfo" pod "93fab1f3-4ba5-46fa-a9f6-001946891f5b" (UID: "93fab1f3-4ba5-46fa-a9f6-001946891f5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:47:58.248306 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.248208 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:58.248306 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.248242 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:58.248306 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.248257 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:58.248306 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.248266 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:58.248306 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.248276 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:58.248306 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.248284 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:58.248306 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.248298 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93fab1f3-4ba5-46fa-a9f6-001946891f5b-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:58.248306 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.248312 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srxjn\" (UniqueName: \"kubernetes.io/projected/93fab1f3-4ba5-46fa-a9f6-001946891f5b-kube-api-access-srxjn\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:47:58.462405 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.462356 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:58.670702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.666490 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:58.740775 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.740738 5490 generic.go:356] "Generic (PLEG): container finished" podID="93fab1f3-4ba5-46fa-a9f6-001946891f5b" containerID="80ceb5748003ba6aad148b3cbff454e941c905cce04047854876fbda303e7923" exitCode=0 Oct 31 03:47:58.741170 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.741148 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" event={"ID":"93fab1f3-4ba5-46fa-a9f6-001946891f5b","Type":"ContainerDied","Data":"80ceb5748003ba6aad148b3cbff454e941c905cce04047854876fbda303e7923"} Oct 31 03:47:58.744234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.741304 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" event={"ID":"93fab1f3-4ba5-46fa-a9f6-001946891f5b","Type":"ContainerDied","Data":"e289c9d17dd8925758bb57e0357c640b3aec9eb77df73fb056a34a936965d464"} Oct 31 03:47:58.744234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.741399 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb" Oct 31 03:47:58.744234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.741433 5490 scope.go:122] "RemoveContainer" containerID="80ceb5748003ba6aad148b3cbff454e941c905cce04047854876fbda303e7923" Oct 31 03:47:58.767858 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.767811 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd"] Oct 31 03:47:58.768760 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.768635 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" podUID="1c1369b1-6901-4b11-a05d-12763e10bf4f" containerName="istio-proxy" containerID="cri-o://e012428bb113329314fc4e7cd260cd3464f5146ed6816ff9d9b6685de55ec84b" gracePeriod=30 Oct 31 03:47:58.795426 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.795395 5490 scope.go:122] "RemoveContainer" containerID="80ceb5748003ba6aad148b3cbff454e941c905cce04047854876fbda303e7923" Oct 31 03:47:58.798840 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:47:58.796377 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80ceb5748003ba6aad148b3cbff454e941c905cce04047854876fbda303e7923\": container with ID starting with 80ceb5748003ba6aad148b3cbff454e941c905cce04047854876fbda303e7923 not found: ID does not exist" containerID="80ceb5748003ba6aad148b3cbff454e941c905cce04047854876fbda303e7923" Oct 31 03:47:58.798840 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.796416 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ceb5748003ba6aad148b3cbff454e941c905cce04047854876fbda303e7923"} err="failed to get container status \"80ceb5748003ba6aad148b3cbff454e941c905cce04047854876fbda303e7923\": rpc error: code = NotFound desc = could not find container \"80ceb5748003ba6aad148b3cbff454e941c905cce04047854876fbda303e7923\": container with ID starting with 80ceb5748003ba6aad148b3cbff454e941c905cce04047854876fbda303e7923 not found: ID does not exist" Oct 31 03:47:58.819733 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.819693 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:58.876707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.873877 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb"] Oct 31 03:47:58.976762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:58.976543 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-8s9nb"] Oct 31 03:47:59.528133 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:59.528100 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" Oct 31 03:47:59.579901 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:59.579864 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93fab1f3-4ba5-46fa-a9f6-001946891f5b" path="/var/lib/kubelet/pods/93fab1f3-4ba5-46fa-a9f6-001946891f5b/volumes" Oct 31 03:47:59.850736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:59.850615 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:47:59.978887 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:59.978846 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr"] Oct 31 03:47:59.979640 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:47:59.979219 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" podUID="680c32cf-d69b-4d52-b696-ca7ecb54e0e9" containerName="istio-proxy" containerID="cri-o://044b1407d9788c38a9ff0fc65a7457b0d542c8e952da7c306fe5bf7627d094df" gracePeriod=30 Oct 31 03:48:00.796695 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:00.796643 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:00.869698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:00.863655 5490 patch_prober.go:29] interesting pod/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:48:00.869698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:00.863830 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" podUID="1c1369b1-6901-4b11-a05d-12763e10bf4f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:48:01.667128 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:01.667088 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:02.325781 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:02.324060 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:03.014224 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:03.014182 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:03.681565 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:03.681521 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:04.108189 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.108151 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:48:04.149446 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.149392 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-data\") pod \"1c1369b1-6901-4b11-a05d-12763e10bf4f\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " Oct 31 03:48:04.150118 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.149512 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c1369b1-6901-4b11-a05d-12763e10bf4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c1369b1-6901-4b11-a05d-12763e10bf4f-istiod-ca-cert\") pod \"1c1369b1-6901-4b11-a05d-12763e10bf4f\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " Oct 31 03:48:04.150118 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.149599 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-podinfo\") pod \"1c1369b1-6901-4b11-a05d-12763e10bf4f\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " Oct 31 03:48:04.150118 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.149657 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-workload-certs\") pod \"1c1369b1-6901-4b11-a05d-12763e10bf4f\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " Oct 31 03:48:04.150118 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.149952 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-data" pod "1c1369b1-6901-4b11-a05d-12763e10bf4f" (UID: "1c1369b1-6901-4b11-a05d-12763e10bf4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:04.150118 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.150085 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-workload-socket\") pod \"1c1369b1-6901-4b11-a05d-12763e10bf4f\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " Oct 31 03:48:04.150528 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.150181 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c1369b1-6901-4b11-a05d-12763e10bf4f-kube-api-access-h4jmk\" (UniqueName: \"kubernetes.io/projected/1c1369b1-6901-4b11-a05d-12763e10bf4f-kube-api-access-h4jmk\") pod \"1c1369b1-6901-4b11-a05d-12763e10bf4f\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " Oct 31 03:48:04.150528 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.150233 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-token\" (UniqueName: \"kubernetes.io/projected/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-token\") pod \"1c1369b1-6901-4b11-a05d-12763e10bf4f\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " Oct 31 03:48:04.150528 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.150490 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-credential-socket\") pod \"1c1369b1-6901-4b11-a05d-12763e10bf4f\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " Oct 31 03:48:04.150852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.150538 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-envoy\") pod \"1c1369b1-6901-4b11-a05d-12763e10bf4f\" (UID: \"1c1369b1-6901-4b11-a05d-12763e10bf4f\") " Oct 31 03:48:04.151869 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.151567 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-workload-certs" pod "1c1369b1-6901-4b11-a05d-12763e10bf4f" (UID: "1c1369b1-6901-4b11-a05d-12763e10bf4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:04.151869 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.151598 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1369b1-6901-4b11-a05d-12763e10bf4f-istiod-ca-cert" pod "1c1369b1-6901-4b11-a05d-12763e10bf4f" (UID: "1c1369b1-6901-4b11-a05d-12763e10bf4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:04.152192 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.152071 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-workload-socket" pod "1c1369b1-6901-4b11-a05d-12763e10bf4f" (UID: "1c1369b1-6901-4b11-a05d-12763e10bf4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:04.153586 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.152586 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-credential-socket" pod "1c1369b1-6901-4b11-a05d-12763e10bf4f" (UID: "1c1369b1-6901-4b11-a05d-12763e10bf4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:04.155808 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.154709 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-envoy" pod "1c1369b1-6901-4b11-a05d-12763e10bf4f" (UID: "1c1369b1-6901-4b11-a05d-12763e10bf4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:04.155808 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.154803 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-token" pod "1c1369b1-6901-4b11-a05d-12763e10bf4f" (UID: "1c1369b1-6901-4b11-a05d-12763e10bf4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:04.156214 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.156122 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:04.156214 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.156152 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:04.156214 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.156166 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:04.156968 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.156456 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c1369b1-6901-4b11-a05d-12763e10bf4f-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:04.156968 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.156482 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:04.156968 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.156496 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c1369b1-6901-4b11-a05d-12763e10bf4f-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:04.156968 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.156510 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:04.156968 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.156633 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-podinfo" pod "1c1369b1-6901-4b11-a05d-12763e10bf4f" (UID: "1c1369b1-6901-4b11-a05d-12763e10bf4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:04.156968 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.156724 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1369b1-6901-4b11-a05d-12763e10bf4f-kube-api-access-h4jmk" pod "1c1369b1-6901-4b11-a05d-12763e10bf4f" (UID: "1c1369b1-6901-4b11-a05d-12763e10bf4f"). InnerVolumeSpecName "kube-api-access-h4jmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:04.258722 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.258606 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c1369b1-6901-4b11-a05d-12763e10bf4f-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:04.258722 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.258688 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4jmk\" (UniqueName: \"kubernetes.io/projected/1c1369b1-6901-4b11-a05d-12763e10bf4f-kube-api-access-h4jmk\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:04.394737 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.394383 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:04.800300 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.800254 5490 generic.go:356] "Generic (PLEG): container finished" podID="1c1369b1-6901-4b11-a05d-12763e10bf4f" containerID="e012428bb113329314fc4e7cd260cd3464f5146ed6816ff9d9b6685de55ec84b" exitCode=0 Oct 31 03:48:04.800909 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.800304 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" Oct 31 03:48:04.800909 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.800315 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" event={"ID":"1c1369b1-6901-4b11-a05d-12763e10bf4f","Type":"ContainerDied","Data":"e012428bb113329314fc4e7cd260cd3464f5146ed6816ff9d9b6685de55ec84b"} Oct 31 03:48:04.800909 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.800373 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd" event={"ID":"1c1369b1-6901-4b11-a05d-12763e10bf4f","Type":"ContainerDied","Data":"3d40810aaff12b111fc3d32906c2d5bdc46dc522e65df4dd3a889e79e98c508e"} Oct 31 03:48:04.800909 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.800403 5490 scope.go:122] "RemoveContainer" containerID="e012428bb113329314fc4e7cd260cd3464f5146ed6816ff9d9b6685de55ec84b" Oct 31 03:48:04.817495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.817469 5490 scope.go:122] "RemoveContainer" containerID="3ac46324dd80274db98b0c7d9e5804c4075a0969b6cf7a5d5a172477c4e31216" Oct 31 03:48:04.828576 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.828539 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd"] Oct 31 03:48:04.831984 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.831958 5490 scope.go:122] "RemoveContainer" containerID="e012428bb113329314fc4e7cd260cd3464f5146ed6816ff9d9b6685de55ec84b" Oct 31 03:48:04.832323 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:48:04.832300 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e012428bb113329314fc4e7cd260cd3464f5146ed6816ff9d9b6685de55ec84b\": container with ID starting with e012428bb113329314fc4e7cd260cd3464f5146ed6816ff9d9b6685de55ec84b not found: ID does not exist" containerID="e012428bb113329314fc4e7cd260cd3464f5146ed6816ff9d9b6685de55ec84b" Oct 31 03:48:04.832422 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.832331 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e012428bb113329314fc4e7cd260cd3464f5146ed6816ff9d9b6685de55ec84b"} err="failed to get container status \"e012428bb113329314fc4e7cd260cd3464f5146ed6816ff9d9b6685de55ec84b\": rpc error: code = NotFound desc = could not find container \"e012428bb113329314fc4e7cd260cd3464f5146ed6816ff9d9b6685de55ec84b\": container with ID starting with e012428bb113329314fc4e7cd260cd3464f5146ed6816ff9d9b6685de55ec84b not found: ID does not exist" Oct 31 03:48:04.832422 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.832350 5490 scope.go:122] "RemoveContainer" containerID="3ac46324dd80274db98b0c7d9e5804c4075a0969b6cf7a5d5a172477c4e31216" Oct 31 03:48:04.832666 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:48:04.832650 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac46324dd80274db98b0c7d9e5804c4075a0969b6cf7a5d5a172477c4e31216\": container with ID starting with 3ac46324dd80274db98b0c7d9e5804c4075a0969b6cf7a5d5a172477c4e31216 not found: ID does not exist" containerID="3ac46324dd80274db98b0c7d9e5804c4075a0969b6cf7a5d5a172477c4e31216" Oct 31 03:48:04.832784 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.832672 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac46324dd80274db98b0c7d9e5804c4075a0969b6cf7a5d5a172477c4e31216"} err="failed to get container status \"3ac46324dd80274db98b0c7d9e5804c4075a0969b6cf7a5d5a172477c4e31216\": rpc error: code = NotFound desc = could not find container \"3ac46324dd80274db98b0c7d9e5804c4075a0969b6cf7a5d5a172477c4e31216\": container with ID starting with 3ac46324dd80274db98b0c7d9e5804c4075a0969b6cf7a5d5a172477c4e31216 not found: ID does not exist" Oct 31 03:48:04.836601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:04.836581 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b2dc82-openshift-default-567d4dbddf-7pgtd"] Oct 31 03:48:05.114863 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.114807 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:05.332847 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.332814 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:48:05.393748 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.393649 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-kube-api-access-j6scn\" (UniqueName: \"kubernetes.io/projected/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-kube-api-access-j6scn\") pod \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " Oct 31 03:48:05.393945 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.393923 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-credential-socket\") pod \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " Oct 31 03:48:05.395230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.393997 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istiod-ca-cert\") pod \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " Oct 31 03:48:05.395230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.394111 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-envoy\") pod \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " Oct 31 03:48:05.395230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.394225 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-token\" (UniqueName: \"kubernetes.io/projected/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-token\") pod \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " Oct 31 03:48:05.395230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.394302 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-data\") pod \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " Oct 31 03:48:05.395230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.394328 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-workload-certs\") pod \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " Oct 31 03:48:05.395230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.394346 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-podinfo\") pod \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " Oct 31 03:48:05.395230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.394416 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-workload-socket\") pod \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\" (UID: \"680c32cf-d69b-4d52-b696-ca7ecb54e0e9\") " Oct 31 03:48:05.395230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.394719 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-credential-socket" pod "680c32cf-d69b-4d52-b696-ca7ecb54e0e9" (UID: "680c32cf-d69b-4d52-b696-ca7ecb54e0e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:05.395230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.394950 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istiod-ca-cert" pod "680c32cf-d69b-4d52-b696-ca7ecb54e0e9" (UID: "680c32cf-d69b-4d52-b696-ca7ecb54e0e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:05.395230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.395037 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-workload-certs" pod "680c32cf-d69b-4d52-b696-ca7ecb54e0e9" (UID: "680c32cf-d69b-4d52-b696-ca7ecb54e0e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:05.395230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.395084 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-data" pod "680c32cf-d69b-4d52-b696-ca7ecb54e0e9" (UID: "680c32cf-d69b-4d52-b696-ca7ecb54e0e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:05.395857 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.395386 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.395857 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.395409 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.395857 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.395426 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.395857 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.395440 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.395857 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.395724 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-workload-socket" pod "680c32cf-d69b-4d52-b696-ca7ecb54e0e9" (UID: "680c32cf-d69b-4d52-b696-ca7ecb54e0e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:05.397140 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.397023 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-kube-api-access-j6scn" pod "680c32cf-d69b-4d52-b696-ca7ecb54e0e9" (UID: "680c32cf-d69b-4d52-b696-ca7ecb54e0e9"). InnerVolumeSpecName "kube-api-access-j6scn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:05.397246 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.397224 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-envoy" pod "680c32cf-d69b-4d52-b696-ca7ecb54e0e9" (UID: "680c32cf-d69b-4d52-b696-ca7ecb54e0e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:05.398214 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.398189 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-podinfo" pod "680c32cf-d69b-4d52-b696-ca7ecb54e0e9" (UID: "680c32cf-d69b-4d52-b696-ca7ecb54e0e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:05.398328 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.398245 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-token" pod "680c32cf-d69b-4d52-b696-ca7ecb54e0e9" (UID: "680c32cf-d69b-4d52-b696-ca7ecb54e0e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:05.497519 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.497473 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6scn\" (UniqueName: \"kubernetes.io/projected/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-kube-api-access-j6scn\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.497519 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.497514 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.497519 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.497527 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.497791 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.497539 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.497791 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.497556 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/680c32cf-d69b-4d52-b696-ca7ecb54e0e9-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:05.579886 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.579852 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c1369b1-6901-4b11-a05d-12763e10bf4f" path="/var/lib/kubelet/pods/1c1369b1-6901-4b11-a05d-12763e10bf4f/volumes" Oct 31 03:48:05.744123 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.743393 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:05.812972 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.812932 5490 generic.go:356] "Generic (PLEG): container finished" podID="680c32cf-d69b-4d52-b696-ca7ecb54e0e9" containerID="044b1407d9788c38a9ff0fc65a7457b0d542c8e952da7c306fe5bf7627d094df" exitCode=0 Oct 31 03:48:05.813584 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.812998 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" Oct 31 03:48:05.813584 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.813033 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" event={"ID":"680c32cf-d69b-4d52-b696-ca7ecb54e0e9","Type":"ContainerDied","Data":"044b1407d9788c38a9ff0fc65a7457b0d542c8e952da7c306fe5bf7627d094df"} Oct 31 03:48:05.813584 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.813104 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr" event={"ID":"680c32cf-d69b-4d52-b696-ca7ecb54e0e9","Type":"ContainerDied","Data":"f16987c8943eef53ffa5ee7bdf8ccf7163ade2100cc1f03db99c6fb8bc75c448"} Oct 31 03:48:05.813584 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.813148 5490 scope.go:122] "RemoveContainer" containerID="044b1407d9788c38a9ff0fc65a7457b0d542c8e952da7c306fe5bf7627d094df" Oct 31 03:48:05.835342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.835303 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr"] Oct 31 03:48:05.840123 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.840099 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-kkbfr"] Oct 31 03:48:05.863235 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.863208 5490 scope.go:122] "RemoveContainer" containerID="044b1407d9788c38a9ff0fc65a7457b0d542c8e952da7c306fe5bf7627d094df" Oct 31 03:48:05.863587 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:48:05.863564 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"044b1407d9788c38a9ff0fc65a7457b0d542c8e952da7c306fe5bf7627d094df\": container with ID starting with 044b1407d9788c38a9ff0fc65a7457b0d542c8e952da7c306fe5bf7627d094df not found: ID does not exist" containerID="044b1407d9788c38a9ff0fc65a7457b0d542c8e952da7c306fe5bf7627d094df" Oct 31 03:48:05.863640 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:05.863597 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"044b1407d9788c38a9ff0fc65a7457b0d542c8e952da7c306fe5bf7627d094df"} err="failed to get container status \"044b1407d9788c38a9ff0fc65a7457b0d542c8e952da7c306fe5bf7627d094df\": rpc error: code = NotFound desc = could not find container \"044b1407d9788c38a9ff0fc65a7457b0d542c8e952da7c306fe5bf7627d094df\": container with ID starting with 044b1407d9788c38a9ff0fc65a7457b0d542c8e952da7c306fe5bf7627d094df not found: ID does not exist" Oct 31 03:48:06.194714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:06.194633 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz"] Oct 31 03:48:06.194914 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:06.194863 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" podUID="77dfaec5-10f9-484c-ad1e-19980c14a3ea" containerName="istio-proxy" containerID="cri-o://b3fca90efe6ceaf5b458e769f9a619be804526b5a65cc08f816a29381af91ea2" gracePeriod=30 Oct 31 03:48:06.382503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:06.374875 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:06.890046 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:06.890012 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz"] Oct 31 03:48:06.890554 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:06.890222 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" podUID="d4568c18-198b-4726-8005-35e3fdbae8d9" containerName="istio-proxy" containerID="cri-o://015ef7b18acfc102e026aa5beba3a3a463b052da244b8d1d08d16fcced18b268" gracePeriod=30 Oct 31 03:48:07.037014 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:07.036977 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2"] Oct 31 03:48:07.037207 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:07.037186 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" podUID="aca6c54e-0673-4a1d-9e32-d653489ec2cf" containerName="istio-proxy" containerID="cri-o://2cd048810fd7457188bb6b9d1083df1b24c2c28593dfe06992dd0137eb0f80d5" gracePeriod=30 Oct 31 03:48:07.073813 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:07.073776 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:07.280939 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:07.280903 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs"] Oct 31 03:48:07.281158 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:07.281133 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" podUID="de21da55-7c0d-4356-9fbb-d077817a9917" containerName="istio-proxy" containerID="cri-o://09dfcaf647aa638d48c6af77846e8efd0eecf08730a021f0795436955fde104f" gracePeriod=30 Oct 31 03:48:07.579840 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:07.579758 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="680c32cf-d69b-4d52-b696-ca7ecb54e0e9" path="/var/lib/kubelet/pods/680c32cf-d69b-4d52-b696-ca7ecb54e0e9/volumes" Oct 31 03:48:07.781615 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:07.781577 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:08.281664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:08.281619 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:08.454705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:08.436143 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:08.565464 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:08.565372 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:08.832739 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:08.832588 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:09.085729 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:09.085569 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:09.565752 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:09.565667 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:09.862275 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:09.862184 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:10.002045 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:10.002010 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:10.792347 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:10.792261 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:11.244109 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:48:11.244064 5490 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77dfaec5_10f9_484c_ad1e_19980c14a3ea.slice/crio-b3fca90efe6ceaf5b458e769f9a619be804526b5a65cc08f816a29381af91ea2.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:48:11.467717 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.461762 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:11.560232 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.560200 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:48:11.663016 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.662978 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-data\") pod \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " Oct 31 03:48:11.663222 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.663062 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-token\" (UniqueName: \"kubernetes.io/projected/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-token\") pod \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " Oct 31 03:48:11.663222 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.663120 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-envoy\") pod \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " Oct 31 03:48:11.663222 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.663152 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-credential-socket\") pod \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " Oct 31 03:48:11.663222 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.663189 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-workload-socket\") pod \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " Oct 31 03:48:11.663435 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.663361 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istiod-ca-cert\") pod \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " Oct 31 03:48:11.663435 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.663376 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-data" pod "77dfaec5-10f9-484c-ad1e-19980c14a3ea" (UID: "77dfaec5-10f9-484c-ad1e-19980c14a3ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:11.663435 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.663396 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77dfaec5-10f9-484c-ad1e-19980c14a3ea-kube-api-access-sjmrj\" (UniqueName: \"kubernetes.io/projected/77dfaec5-10f9-484c-ad1e-19980c14a3ea-kube-api-access-sjmrj\") pod \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " Oct 31 03:48:11.663585 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.663442 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-workload-certs\") pod \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " Oct 31 03:48:11.663585 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.663523 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-podinfo\") pod \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\" (UID: \"77dfaec5-10f9-484c-ad1e-19980c14a3ea\") " Oct 31 03:48:11.663845 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.663814 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-workload-socket" pod "77dfaec5-10f9-484c-ad1e-19980c14a3ea" (UID: "77dfaec5-10f9-484c-ad1e-19980c14a3ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:11.663933 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.663871 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istiod-ca-cert" pod "77dfaec5-10f9-484c-ad1e-19980c14a3ea" (UID: "77dfaec5-10f9-484c-ad1e-19980c14a3ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:11.664595 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.664564 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-credential-socket" pod "77dfaec5-10f9-484c-ad1e-19980c14a3ea" (UID: "77dfaec5-10f9-484c-ad1e-19980c14a3ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:11.664824 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.664589 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-workload-certs" pod "77dfaec5-10f9-484c-ad1e-19980c14a3ea" (UID: "77dfaec5-10f9-484c-ad1e-19980c14a3ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:11.665072 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.665043 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:11.665159 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.665080 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:11.665159 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.665096 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:11.665159 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.665111 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:11.665159 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.665125 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:11.667272 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.666569 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-token" pod "77dfaec5-10f9-484c-ad1e-19980c14a3ea" (UID: "77dfaec5-10f9-484c-ad1e-19980c14a3ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:11.667272 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.667032 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-envoy" pod "77dfaec5-10f9-484c-ad1e-19980c14a3ea" (UID: "77dfaec5-10f9-484c-ad1e-19980c14a3ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:11.667272 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.667255 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77dfaec5-10f9-484c-ad1e-19980c14a3ea-kube-api-access-sjmrj" pod "77dfaec5-10f9-484c-ad1e-19980c14a3ea" (UID: "77dfaec5-10f9-484c-ad1e-19980c14a3ea"). InnerVolumeSpecName "kube-api-access-sjmrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:11.667424 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.667338 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-podinfo" pod "77dfaec5-10f9-484c-ad1e-19980c14a3ea" (UID: "77dfaec5-10f9-484c-ad1e-19980c14a3ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:11.766956 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.766923 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:11.766956 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.766953 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:11.767190 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.766966 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjmrj\" (UniqueName: \"kubernetes.io/projected/77dfaec5-10f9-484c-ad1e-19980c14a3ea-kube-api-access-sjmrj\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:11.767190 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.766977 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77dfaec5-10f9-484c-ad1e-19980c14a3ea-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:11.954307 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.954265 5490 generic.go:356] "Generic (PLEG): container finished" podID="77dfaec5-10f9-484c-ad1e-19980c14a3ea" containerID="b3fca90efe6ceaf5b458e769f9a619be804526b5a65cc08f816a29381af91ea2" exitCode=0 Oct 31 03:48:11.954807 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.954331 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" Oct 31 03:48:11.954807 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.954344 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" event={"ID":"77dfaec5-10f9-484c-ad1e-19980c14a3ea","Type":"ContainerDied","Data":"b3fca90efe6ceaf5b458e769f9a619be804526b5a65cc08f816a29381af91ea2"} Oct 31 03:48:11.954807 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.954419 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz" event={"ID":"77dfaec5-10f9-484c-ad1e-19980c14a3ea","Type":"ContainerDied","Data":"c074339ec2f4e3d02772718a3a6921abe88675e28ca1843ceb27cf9b8b3f53cb"} Oct 31 03:48:11.954807 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.954447 5490 scope.go:122] "RemoveContainer" containerID="b3fca90efe6ceaf5b458e769f9a619be804526b5a65cc08f816a29381af91ea2" Oct 31 03:48:11.992373 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:11.992339 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz"] Oct 31 03:48:12.002058 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.002024 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-rfpkz"] Oct 31 03:48:12.007997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.007975 5490 scope.go:122] "RemoveContainer" containerID="b3fca90efe6ceaf5b458e769f9a619be804526b5a65cc08f816a29381af91ea2" Oct 31 03:48:12.008653 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:48:12.008305 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3fca90efe6ceaf5b458e769f9a619be804526b5a65cc08f816a29381af91ea2\": container with ID starting with b3fca90efe6ceaf5b458e769f9a619be804526b5a65cc08f816a29381af91ea2 not found: ID does not exist" containerID="b3fca90efe6ceaf5b458e769f9a619be804526b5a65cc08f816a29381af91ea2" Oct 31 03:48:12.008653 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.008332 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3fca90efe6ceaf5b458e769f9a619be804526b5a65cc08f816a29381af91ea2"} err="failed to get container status \"b3fca90efe6ceaf5b458e769f9a619be804526b5a65cc08f816a29381af91ea2\": rpc error: code = NotFound desc = could not find container \"b3fca90efe6ceaf5b458e769f9a619be804526b5a65cc08f816a29381af91ea2\": container with ID starting with b3fca90efe6ceaf5b458e769f9a619be804526b5a65cc08f816a29381af91ea2 not found: ID does not exist" Oct 31 03:48:12.196285 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.196182 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:12.216984 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.216957 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:48:12.404913 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.404882 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-envoy\") pod \"d4568c18-198b-4726-8005-35e3fdbae8d9\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " Oct 31 03:48:12.405102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.405066 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-credential-socket\") pod \"d4568c18-198b-4726-8005-35e3fdbae8d9\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " Oct 31 03:48:12.406544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.405243 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4568c18-198b-4726-8005-35e3fdbae8d9-kube-api-access-2zczv\" (UniqueName: \"kubernetes.io/projected/d4568c18-198b-4726-8005-35e3fdbae8d9-kube-api-access-2zczv\") pod \"d4568c18-198b-4726-8005-35e3fdbae8d9\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " Oct 31 03:48:12.406544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.405288 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-workload-socket\") pod \"d4568c18-198b-4726-8005-35e3fdbae8d9\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " Oct 31 03:48:12.406544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.405405 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-data\") pod \"d4568c18-198b-4726-8005-35e3fdbae8d9\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " Oct 31 03:48:12.406544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.405432 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-credential-socket" pod "d4568c18-198b-4726-8005-35e3fdbae8d9" (UID: "d4568c18-198b-4726-8005-35e3fdbae8d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:12.406544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.405457 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-podinfo\") pod \"d4568c18-198b-4726-8005-35e3fdbae8d9\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " Oct 31 03:48:12.406544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.405534 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-token\" (UniqueName: \"kubernetes.io/projected/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-token\") pod \"d4568c18-198b-4726-8005-35e3fdbae8d9\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " Oct 31 03:48:12.406544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.405568 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4568c18-198b-4726-8005-35e3fdbae8d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4568c18-198b-4726-8005-35e3fdbae8d9-istiod-ca-cert\") pod \"d4568c18-198b-4726-8005-35e3fdbae8d9\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " Oct 31 03:48:12.406544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.405621 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-workload-certs\") pod \"d4568c18-198b-4726-8005-35e3fdbae8d9\" (UID: \"d4568c18-198b-4726-8005-35e3fdbae8d9\") " Oct 31 03:48:12.406544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.406282 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.407613 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.407022 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-workload-socket" pod "d4568c18-198b-4726-8005-35e3fdbae8d9" (UID: "d4568c18-198b-4726-8005-35e3fdbae8d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:12.407613 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.407036 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-data" pod "d4568c18-198b-4726-8005-35e3fdbae8d9" (UID: "d4568c18-198b-4726-8005-35e3fdbae8d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:12.407613 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.407360 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4568c18-198b-4726-8005-35e3fdbae8d9-istiod-ca-cert" pod "d4568c18-198b-4726-8005-35e3fdbae8d9" (UID: "d4568c18-198b-4726-8005-35e3fdbae8d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:12.408451 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.408267 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-workload-certs" pod "d4568c18-198b-4726-8005-35e3fdbae8d9" (UID: "d4568c18-198b-4726-8005-35e3fdbae8d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:12.410870 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.410836 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-envoy" pod "d4568c18-198b-4726-8005-35e3fdbae8d9" (UID: "d4568c18-198b-4726-8005-35e3fdbae8d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:12.410870 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.410852 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-podinfo" pod "d4568c18-198b-4726-8005-35e3fdbae8d9" (UID: "d4568c18-198b-4726-8005-35e3fdbae8d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:12.411026 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.410886 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-token" pod "d4568c18-198b-4726-8005-35e3fdbae8d9" (UID: "d4568c18-198b-4726-8005-35e3fdbae8d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:12.411026 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.410917 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4568c18-198b-4726-8005-35e3fdbae8d9-kube-api-access-2zczv" pod "d4568c18-198b-4726-8005-35e3fdbae8d9" (UID: "d4568c18-198b-4726-8005-35e3fdbae8d9"). InnerVolumeSpecName "kube-api-access-2zczv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:12.443866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.443726 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:48:12.508611 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.508566 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zczv\" (UniqueName: \"kubernetes.io/projected/d4568c18-198b-4726-8005-35e3fdbae8d9-kube-api-access-2zczv\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.508611 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.508614 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.508896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.508630 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.508896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.508643 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.508896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.508660 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.508896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.508690 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4568c18-198b-4726-8005-35e3fdbae8d9-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.508896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.508704 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.508896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.508719 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4568c18-198b-4726-8005-35e3fdbae8d9-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.570561 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.570526 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:48:12.609973 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.609927 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-credential-socket\") pod \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " Oct 31 03:48:12.610765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.609985 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-envoy\") pod \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " Oct 31 03:48:12.610765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.610028 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-data\") pod \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " Oct 31 03:48:12.610765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.610125 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca6c54e-0673-4a1d-9e32-d653489ec2cf-kube-api-access-mgjmd\" (UniqueName: \"kubernetes.io/projected/aca6c54e-0673-4a1d-9e32-d653489ec2cf-kube-api-access-mgjmd\") pod \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " Oct 31 03:48:12.610765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.610196 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-token\" (UniqueName: \"kubernetes.io/projected/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-token\") pod \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " Oct 31 03:48:12.610765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.610262 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-podinfo\") pod \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " Oct 31 03:48:12.610765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.610288 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-credential-socket" pod "aca6c54e-0673-4a1d-9e32-d653489ec2cf" (UID: "aca6c54e-0673-4a1d-9e32-d653489ec2cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:12.610765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.610399 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-data" pod "aca6c54e-0673-4a1d-9e32-d653489ec2cf" (UID: "aca6c54e-0673-4a1d-9e32-d653489ec2cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:12.610765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.610443 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istiod-ca-cert\") pod \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " Oct 31 03:48:12.610765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.610495 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-workload-socket\") pod \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " Oct 31 03:48:12.610765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.610642 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-workload-certs\") pod \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\" (UID: \"aca6c54e-0673-4a1d-9e32-d653489ec2cf\") " Oct 31 03:48:12.612371 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.611009 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-workload-socket" pod "aca6c54e-0673-4a1d-9e32-d653489ec2cf" (UID: "aca6c54e-0673-4a1d-9e32-d653489ec2cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:12.612371 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.611044 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-workload-certs" pod "aca6c54e-0673-4a1d-9e32-d653489ec2cf" (UID: "aca6c54e-0673-4a1d-9e32-d653489ec2cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:12.612371 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.611394 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istiod-ca-cert" pod "aca6c54e-0673-4a1d-9e32-d653489ec2cf" (UID: "aca6c54e-0673-4a1d-9e32-d653489ec2cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:12.612929 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.612907 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.613062 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.613048 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.613182 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.613171 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.613269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.613258 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.613349 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.613338 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.613636 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.613580 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca6c54e-0673-4a1d-9e32-d653489ec2cf-kube-api-access-mgjmd" pod "aca6c54e-0673-4a1d-9e32-d653489ec2cf" (UID: "aca6c54e-0673-4a1d-9e32-d653489ec2cf"). InnerVolumeSpecName "kube-api-access-mgjmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:12.614989 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.614953 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-token" pod "aca6c54e-0673-4a1d-9e32-d653489ec2cf" (UID: "aca6c54e-0673-4a1d-9e32-d653489ec2cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:12.616240 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.616213 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-envoy" pod "aca6c54e-0673-4a1d-9e32-d653489ec2cf" (UID: "aca6c54e-0673-4a1d-9e32-d653489ec2cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:12.616423 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.616403 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-podinfo" pod "aca6c54e-0673-4a1d-9e32-d653489ec2cf" (UID: "aca6c54e-0673-4a1d-9e32-d653489ec2cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:12.714291 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.714247 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-istio-envoy\") pod \"de21da55-7c0d-4356-9fbb-d077817a9917\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " Oct 31 03:48:12.715801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.714365 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-workload-socket\") pod \"de21da55-7c0d-4356-9fbb-d077817a9917\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " Oct 31 03:48:12.715801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.714454 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-credential-socket\") pod \"de21da55-7c0d-4356-9fbb-d077817a9917\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " Oct 31 03:48:12.715801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.714542 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-istio-data\") pod \"de21da55-7c0d-4356-9fbb-d077817a9917\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " Oct 31 03:48:12.715801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.714630 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-workload-certs\") pod \"de21da55-7c0d-4356-9fbb-d077817a9917\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " Oct 31 03:48:12.715801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.714653 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-workload-socket" pod "de21da55-7c0d-4356-9fbb-d077817a9917" (UID: "de21da55-7c0d-4356-9fbb-d077817a9917"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:12.715801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.714714 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de21da55-7c0d-4356-9fbb-d077817a9917-kube-api-access-7z7hk\" (UniqueName: \"kubernetes.io/projected/de21da55-7c0d-4356-9fbb-d077817a9917-kube-api-access-7z7hk\") pod \"de21da55-7c0d-4356-9fbb-d077817a9917\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " Oct 31 03:48:12.715801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.714756 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de21da55-7c0d-4356-9fbb-d077817a9917-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de21da55-7c0d-4356-9fbb-d077817a9917-istiod-ca-cert\") pod \"de21da55-7c0d-4356-9fbb-d077817a9917\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " Oct 31 03:48:12.715801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.714837 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de21da55-7c0d-4356-9fbb-d077817a9917-istio-token\" (UniqueName: \"kubernetes.io/projected/de21da55-7c0d-4356-9fbb-d077817a9917-istio-token\") pod \"de21da55-7c0d-4356-9fbb-d077817a9917\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " Oct 31 03:48:12.715801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.714875 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de21da55-7c0d-4356-9fbb-d077817a9917-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de21da55-7c0d-4356-9fbb-d077817a9917-istio-podinfo\") pod \"de21da55-7c0d-4356-9fbb-d077817a9917\" (UID: \"de21da55-7c0d-4356-9fbb-d077817a9917\") " Oct 31 03:48:12.716226 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.715878 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.716226 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.715897 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.716226 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.715908 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgjmd\" (UniqueName: \"kubernetes.io/projected/aca6c54e-0673-4a1d-9e32-d653489ec2cf-kube-api-access-mgjmd\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.716226 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.715918 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.716226 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.715926 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca6c54e-0673-4a1d-9e32-d653489ec2cf-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.716655 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.716605 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-workload-certs" pod "de21da55-7c0d-4356-9fbb-d077817a9917" (UID: "de21da55-7c0d-4356-9fbb-d077817a9917"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:12.716844 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.716820 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de21da55-7c0d-4356-9fbb-d077817a9917-istiod-ca-cert" pod "de21da55-7c0d-4356-9fbb-d077817a9917" (UID: "de21da55-7c0d-4356-9fbb-d077817a9917"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:12.717139 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.717072 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-credential-socket" pod "de21da55-7c0d-4356-9fbb-d077817a9917" (UID: "de21da55-7c0d-4356-9fbb-d077817a9917"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:12.718202 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.718181 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-istio-data" pod "de21da55-7c0d-4356-9fbb-d077817a9917" (UID: "de21da55-7c0d-4356-9fbb-d077817a9917"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:12.718724 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.718691 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-istio-envoy" pod "de21da55-7c0d-4356-9fbb-d077817a9917" (UID: "de21da55-7c0d-4356-9fbb-d077817a9917"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:12.720848 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.720809 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de21da55-7c0d-4356-9fbb-d077817a9917-istio-token" pod "de21da55-7c0d-4356-9fbb-d077817a9917" (UID: "de21da55-7c0d-4356-9fbb-d077817a9917"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:12.722331 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.722306 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de21da55-7c0d-4356-9fbb-d077817a9917-istio-podinfo" pod "de21da55-7c0d-4356-9fbb-d077817a9917" (UID: "de21da55-7c0d-4356-9fbb-d077817a9917"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:12.722611 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.722590 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de21da55-7c0d-4356-9fbb-d077817a9917-kube-api-access-7z7hk" pod "de21da55-7c0d-4356-9fbb-d077817a9917" (UID: "de21da55-7c0d-4356-9fbb-d077817a9917"). InnerVolumeSpecName "kube-api-access-7z7hk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:12.817959 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.817832 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.817959 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.817916 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.817959 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.817933 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.817959 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.817948 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de21da55-7c0d-4356-9fbb-d077817a9917-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.818213 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.817962 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z7hk\" (UniqueName: \"kubernetes.io/projected/de21da55-7c0d-4356-9fbb-d077817a9917-kube-api-access-7z7hk\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.818213 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.817979 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de21da55-7c0d-4356-9fbb-d077817a9917-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.818213 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.817994 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de21da55-7c0d-4356-9fbb-d077817a9917-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:12.818213 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:12.818008 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de21da55-7c0d-4356-9fbb-d077817a9917-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:13.016698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.010149 5490 generic.go:356] "Generic (PLEG): container finished" podID="de21da55-7c0d-4356-9fbb-d077817a9917" containerID="09dfcaf647aa638d48c6af77846e8efd0eecf08730a021f0795436955fde104f" exitCode=0 Oct 31 03:48:13.016698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.010295 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" event={"ID":"de21da55-7c0d-4356-9fbb-d077817a9917","Type":"ContainerDied","Data":"09dfcaf647aa638d48c6af77846e8efd0eecf08730a021f0795436955fde104f"} Oct 31 03:48:13.016698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.010370 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" event={"ID":"de21da55-7c0d-4356-9fbb-d077817a9917","Type":"ContainerDied","Data":"4cf7e95a6a4c43321ccd3dc5e7ae0f460bb4ed0defc31a2b929eea391cc414fe"} Oct 31 03:48:13.016698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.010419 5490 scope.go:122] "RemoveContainer" containerID="09dfcaf647aa638d48c6af77846e8efd0eecf08730a021f0795436955fde104f" Oct 31 03:48:13.016698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.010583 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs" Oct 31 03:48:13.054531 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.052164 5490 generic.go:356] "Generic (PLEG): container finished" podID="aca6c54e-0673-4a1d-9e32-d653489ec2cf" containerID="2cd048810fd7457188bb6b9d1083df1b24c2c28593dfe06992dd0137eb0f80d5" exitCode=0 Oct 31 03:48:13.054531 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.052358 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" event={"ID":"aca6c54e-0673-4a1d-9e32-d653489ec2cf","Type":"ContainerDied","Data":"2cd048810fd7457188bb6b9d1083df1b24c2c28593dfe06992dd0137eb0f80d5"} Oct 31 03:48:13.054531 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.052416 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" event={"ID":"aca6c54e-0673-4a1d-9e32-d653489ec2cf","Type":"ContainerDied","Data":"3c115cd8f7f47bff5bf00cd1d562eacf41cdf5739584bf09e34bb2176886dd58"} Oct 31 03:48:13.054531 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.052501 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2" Oct 31 03:48:13.092953 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.086590 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7"] Oct 31 03:48:13.127645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.118038 5490 generic.go:356] "Generic (PLEG): container finished" podID="d4568c18-198b-4726-8005-35e3fdbae8d9" containerID="015ef7b18acfc102e026aa5beba3a3a463b052da244b8d1d08d16fcced18b268" exitCode=0 Oct 31 03:48:13.128034 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.118110 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" Oct 31 03:48:13.152652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.152622 5490 cpu_manager.go:414] "containerMap: removing container" podUID="1c1369b1-6901-4b11-a05d-12763e10bf4f" containerName="istio-proxy" Oct 31 03:48:13.153000 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.152890 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c1369b1-6901-4b11-a05d-12763e10bf4f" containerName="istio-proxy" Oct 31 03:48:13.153131 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.153119 5490 cpu_manager.go:414] "containerMap: removing container" podUID="77dfaec5-10f9-484c-ad1e-19980c14a3ea" containerName="istio-proxy" Oct 31 03:48:13.154462 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.154438 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77dfaec5-10f9-484c-ad1e-19980c14a3ea" containerName="istio-proxy" Oct 31 03:48:13.154927 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.154912 5490 cpu_manager.go:414] "containerMap: removing container" podUID="93fab1f3-4ba5-46fa-a9f6-001946891f5b" containerName="istio-proxy" Oct 31 03:48:13.155042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.155031 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93fab1f3-4ba5-46fa-a9f6-001946891f5b" containerName="istio-proxy" Oct 31 03:48:13.155145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.155135 5490 cpu_manager.go:414] "containerMap: removing container" podUID="aca6c54e-0673-4a1d-9e32-d653489ec2cf" containerName="istio-proxy" Oct 31 03:48:13.155806 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.155226 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca6c54e-0673-4a1d-9e32-d653489ec2cf" containerName="istio-proxy" Oct 31 03:48:13.155974 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.155956 5490 cpu_manager.go:414] "containerMap: removing container" podUID="de21da55-7c0d-4356-9fbb-d077817a9917" containerName="istio-proxy" Oct 31 03:48:13.156062 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.156052 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de21da55-7c0d-4356-9fbb-d077817a9917" containerName="istio-proxy" Oct 31 03:48:13.156815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.156800 5490 cpu_manager.go:414] "containerMap: removing container" podUID="d4568c18-198b-4726-8005-35e3fdbae8d9" containerName="istio-proxy" Oct 31 03:48:13.156938 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.156926 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4568c18-198b-4726-8005-35e3fdbae8d9" containerName="istio-proxy" Oct 31 03:48:13.157045 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.157036 5490 cpu_manager.go:414] "containerMap: removing container" podUID="680c32cf-d69b-4d52-b696-ca7ecb54e0e9" containerName="istio-proxy" Oct 31 03:48:13.157119 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.157110 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680c32cf-d69b-4d52-b696-ca7ecb54e0e9" containerName="istio-proxy" Oct 31 03:48:13.157194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.157183 5490 cpu_manager.go:414] "containerMap: removing container" podUID="1c1369b1-6901-4b11-a05d-12763e10bf4f" containerName="istio-proxy" Oct 31 03:48:13.157291 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.157278 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c1369b1-6901-4b11-a05d-12763e10bf4f" containerName="istio-proxy" Oct 31 03:48:13.157650 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.157628 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="de21da55-7c0d-4356-9fbb-d077817a9917" containerName="istio-proxy" Oct 31 03:48:13.160859 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.160840 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4568c18-198b-4726-8005-35e3fdbae8d9" containerName="istio-proxy" Oct 31 03:48:13.160972 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.160963 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="680c32cf-d69b-4d52-b696-ca7ecb54e0e9" containerName="istio-proxy" Oct 31 03:48:13.161054 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.161045 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c1369b1-6901-4b11-a05d-12763e10bf4f" containerName="istio-proxy" Oct 31 03:48:13.161154 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.161140 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="77dfaec5-10f9-484c-ad1e-19980c14a3ea" containerName="istio-proxy" Oct 31 03:48:13.181941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.181912 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="93fab1f3-4ba5-46fa-a9f6-001946891f5b" containerName="istio-proxy" Oct 31 03:48:13.182403 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.182383 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca6c54e-0673-4a1d-9e32-d653489ec2cf" containerName="istio-proxy" Oct 31 03:48:13.200007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.199964 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7"] Oct 31 03:48:13.200235 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.200221 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2"] Oct 31 03:48:13.200371 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.200350 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" event={"ID":"d4568c18-198b-4726-8005-35e3fdbae8d9","Type":"ContainerDied","Data":"015ef7b18acfc102e026aa5beba3a3a463b052da244b8d1d08d16fcced18b268"} Oct 31 03:48:13.200561 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.200534 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-8qkx2"] Oct 31 03:48:13.201186 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.200696 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz" event={"ID":"d4568c18-198b-4726-8005-35e3fdbae8d9","Type":"ContainerDied","Data":"7ade8bb6cc5966415a4feba3509d2178cdfac20d7b96283ba938a031fb227318"} Oct 31 03:48:13.201186 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.200738 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs"] Oct 31 03:48:13.201186 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.200890 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" Oct 31 03:48:13.210202 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.207615 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ce9b89-serving-cert\"" Oct 31 03:48:13.215924 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.215895 5490 scope.go:122] "RemoveContainer" containerID="09dfcaf647aa638d48c6af77846e8efd0eecf08730a021f0795436955fde104f" Oct 31 03:48:13.221505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.221033 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6ffzs"] Oct 31 03:48:13.228245 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:48:13.227238 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09dfcaf647aa638d48c6af77846e8efd0eecf08730a021f0795436955fde104f\": container with ID starting with 09dfcaf647aa638d48c6af77846e8efd0eecf08730a021f0795436955fde104f not found: ID does not exist" containerID="09dfcaf647aa638d48c6af77846e8efd0eecf08730a021f0795436955fde104f" Oct 31 03:48:13.228245 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.227282 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09dfcaf647aa638d48c6af77846e8efd0eecf08730a021f0795436955fde104f"} err="failed to get container status \"09dfcaf647aa638d48c6af77846e8efd0eecf08730a021f0795436955fde104f\": rpc error: code = NotFound desc = could not find container \"09dfcaf647aa638d48c6af77846e8efd0eecf08730a021f0795436955fde104f\": container with ID starting with 09dfcaf647aa638d48c6af77846e8efd0eecf08730a021f0795436955fde104f not found: ID does not exist" Oct 31 03:48:13.228245 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.227310 5490 scope.go:122] "RemoveContainer" containerID="2cd048810fd7457188bb6b9d1083df1b24c2c28593dfe06992dd0137eb0f80d5" Oct 31 03:48:13.229516 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.229092 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:13.303068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.302666 5490 scope.go:122] "RemoveContainer" containerID="2cd048810fd7457188bb6b9d1083df1b24c2c28593dfe06992dd0137eb0f80d5" Oct 31 03:48:13.315599 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:48:13.314444 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd048810fd7457188bb6b9d1083df1b24c2c28593dfe06992dd0137eb0f80d5\": container with ID starting with 2cd048810fd7457188bb6b9d1083df1b24c2c28593dfe06992dd0137eb0f80d5 not found: ID does not exist" containerID="2cd048810fd7457188bb6b9d1083df1b24c2c28593dfe06992dd0137eb0f80d5" Oct 31 03:48:13.315599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.314563 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd048810fd7457188bb6b9d1083df1b24c2c28593dfe06992dd0137eb0f80d5"} err="failed to get container status \"2cd048810fd7457188bb6b9d1083df1b24c2c28593dfe06992dd0137eb0f80d5\": rpc error: code = NotFound desc = could not find container \"2cd048810fd7457188bb6b9d1083df1b24c2c28593dfe06992dd0137eb0f80d5\": container with ID starting with 2cd048810fd7457188bb6b9d1083df1b24c2c28593dfe06992dd0137eb0f80d5 not found: ID does not exist" Oct 31 03:48:13.315599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.314592 5490 scope.go:122] "RemoveContainer" containerID="015ef7b18acfc102e026aa5beba3a3a463b052da244b8d1d08d16fcced18b268" Oct 31 03:48:13.326142 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.326020 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz"] Oct 31 03:48:13.334192 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.333255 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-4fxbz"] Oct 31 03:48:13.334192 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.333438 5490 scope.go:122] "RemoveContainer" containerID="015ef7b18acfc102e026aa5beba3a3a463b052da244b8d1d08d16fcced18b268" Oct 31 03:48:13.334192 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:48:13.333935 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"015ef7b18acfc102e026aa5beba3a3a463b052da244b8d1d08d16fcced18b268\": container with ID starting with 015ef7b18acfc102e026aa5beba3a3a463b052da244b8d1d08d16fcced18b268 not found: ID does not exist" containerID="015ef7b18acfc102e026aa5beba3a3a463b052da244b8d1d08d16fcced18b268" Oct 31 03:48:13.334192 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.333971 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"015ef7b18acfc102e026aa5beba3a3a463b052da244b8d1d08d16fcced18b268"} err="failed to get container status \"015ef7b18acfc102e026aa5beba3a3a463b052da244b8d1d08d16fcced18b268\": rpc error: code = NotFound desc = could not find container \"015ef7b18acfc102e026aa5beba3a3a463b052da244b8d1d08d16fcced18b268\": container with ID starting with 015ef7b18acfc102e026aa5beba3a3a463b052da244b8d1d08d16fcced18b268 not found: ID does not exist" Oct 31 03:48:13.354175 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.354123 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:13.362093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.362060 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x854h\" (UniqueName: \"kubernetes.io/projected/253719d9-0fac-4d41-bfc3-d4d359ba04fe-kube-api-access-x854h\") pod \"maas-api-e2e-inf-b-ce9b89-5745885799-w28x7\" (UID: \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" Oct 31 03:48:13.362284 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.362177 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/253719d9-0fac-4d41-bfc3-d4d359ba04fe-metrics-certs\") pod \"maas-api-e2e-inf-b-ce9b89-5745885799-w28x7\" (UID: \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" Oct 31 03:48:13.362357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.362299 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/253719d9-0fac-4d41-bfc3-d4d359ba04fe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ce9b89-5745885799-w28x7\" (UID: \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" Oct 31 03:48:13.362414 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.362389 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/253719d9-0fac-4d41-bfc3-d4d359ba04fe-maas-api-tls\") pod \"maas-api-e2e-inf-b-ce9b89-5745885799-w28x7\" (UID: \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" Oct 31 03:48:13.464262 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.464226 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/253719d9-0fac-4d41-bfc3-d4d359ba04fe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ce9b89-5745885799-w28x7\" (UID: \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" Oct 31 03:48:13.464459 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.464304 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/253719d9-0fac-4d41-bfc3-d4d359ba04fe-maas-api-tls\") pod \"maas-api-e2e-inf-b-ce9b89-5745885799-w28x7\" (UID: \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" Oct 31 03:48:13.464459 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.464453 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x854h\" (UniqueName: \"kubernetes.io/projected/253719d9-0fac-4d41-bfc3-d4d359ba04fe-kube-api-access-x854h\") pod \"maas-api-e2e-inf-b-ce9b89-5745885799-w28x7\" (UID: \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" Oct 31 03:48:13.464617 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.464515 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/253719d9-0fac-4d41-bfc3-d4d359ba04fe-metrics-certs\") pod \"maas-api-e2e-inf-b-ce9b89-5745885799-w28x7\" (UID: \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" Oct 31 03:48:13.465437 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.465416 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/253719d9-0fac-4d41-bfc3-d4d359ba04fe-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ce9b89-5745885799-w28x7\" (UID: \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" Oct 31 03:48:13.465558 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:48:13.465525 5490 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-serving-cert: secret "maas-api-e2e-inf-b-ce9b89-serving-cert" not found Oct 31 03:48:13.466798 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:48:13.465582 5490 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/253719d9-0fac-4d41-bfc3-d4d359ba04fe-maas-api-tls podName:253719d9-0fac-4d41-bfc3-d4d359ba04fe nodeName:}" failed. No retries permitted until 2026-10-31 03:48:13.965561671 +0000 UTC m=+1928.500256604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/253719d9-0fac-4d41-bfc3-d4d359ba04fe-maas-api-tls") pod "maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" (UID: "253719d9-0fac-4d41-bfc3-d4d359ba04fe") : secret "maas-api-e2e-inf-b-ce9b89-serving-cert" not found Oct 31 03:48:13.469092 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.469067 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/253719d9-0fac-4d41-bfc3-d4d359ba04fe-metrics-certs\") pod \"maas-api-e2e-inf-b-ce9b89-5745885799-w28x7\" (UID: \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" Oct 31 03:48:13.476233 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.476199 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x854h\" (UniqueName: \"kubernetes.io/projected/253719d9-0fac-4d41-bfc3-d4d359ba04fe-kube-api-access-x854h\") pod \"maas-api-e2e-inf-b-ce9b89-5745885799-w28x7\" (UID: \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" Oct 31 03:48:13.578360 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.578279 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77dfaec5-10f9-484c-ad1e-19980c14a3ea" path="/var/lib/kubelet/pods/77dfaec5-10f9-484c-ad1e-19980c14a3ea/volumes" Oct 31 03:48:13.578893 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.578878 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aca6c54e-0673-4a1d-9e32-d653489ec2cf" path="/var/lib/kubelet/pods/aca6c54e-0673-4a1d-9e32-d653489ec2cf/volumes" Oct 31 03:48:13.579931 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.579914 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4568c18-198b-4726-8005-35e3fdbae8d9" path="/var/lib/kubelet/pods/d4568c18-198b-4726-8005-35e3fdbae8d9/volumes" Oct 31 03:48:13.580540 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.580517 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de21da55-7c0d-4356-9fbb-d077817a9917" path="/var/lib/kubelet/pods/de21da55-7c0d-4356-9fbb-d077817a9917/volumes" Oct 31 03:48:13.581447 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:13.581421 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:14.108261 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:14.108208 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/253719d9-0fac-4d41-bfc3-d4d359ba04fe-maas-api-tls\") pod \"maas-api-e2e-inf-b-ce9b89-5745885799-w28x7\" (UID: \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" Oct 31 03:48:14.125181 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:14.125140 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/253719d9-0fac-4d41-bfc3-d4d359ba04fe-maas-api-tls\") pod \"maas-api-e2e-inf-b-ce9b89-5745885799-w28x7\" (UID: \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" Oct 31 03:48:14.214170 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:14.214129 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" Oct 31 03:48:14.376779 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:14.376633 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:14.569775 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:14.569744 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7"] Oct 31 03:48:14.570089 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:48:14.570062 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253719d9_0fac_4d41_bfc3_d4d359ba04fe.slice/crio-249edabfff48031be2a97babfb8fc95280527b8bfa22cd703eb79f1188bcbb19 WatchSource:0}: Error finding container 249edabfff48031be2a97babfb8fc95280527b8bfa22cd703eb79f1188bcbb19: Status 404 returned error can't find the container with id 249edabfff48031be2a97babfb8fc95280527b8bfa22cd703eb79f1188bcbb19 Oct 31 03:48:14.924905 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:14.924867 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:15.236260 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:15.236222 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" event={"ID":"253719d9-0fac-4d41-bfc3-d4d359ba04fe","Type":"ContainerStarted","Data":"249edabfff48031be2a97babfb8fc95280527b8bfa22cd703eb79f1188bcbb19"} Oct 31 03:48:15.417664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:15.417576 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:15.567012 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:15.566912 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:15.708615 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:15.707283 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:15.888577 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:15.888479 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:16.251736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:16.251694 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" event={"ID":"253719d9-0fac-4d41-bfc3-d4d359ba04fe","Type":"ContainerStarted","Data":"46fbe8f11e4bf3fd2ead6aa0fbca1a5a6ae4a0e07d6ee6ea419473607dbf493e"} Oct 31 03:48:16.252224 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:16.251965 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" Oct 31 03:48:16.277731 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:16.277659 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" podStartSLOduration=2.8601039740000003 podStartE2EDuration="3.277645177s" totalImagesPullingTime="417.541203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:16.270868637 +0000 UTC m=+1930.805563594" watchObservedRunningTime="2026-10-31 03:48:16.277645177 +0000 UTC m=+1930.812340132" Oct 31 03:48:16.645065 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:16.644971 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:17.135202 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:48:17.135158 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:48:17.429726 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:17.429685 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:17.572431 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:17.572399 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:17.737047 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:17.736956 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:17.863427 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:17.863388 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:18.191660 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.191623 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:18.253936 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.253869 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996"] Oct 31 03:48:18.261046 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.258060 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c1369b1-6901-4b11-a05d-12763e10bf4f" containerName="istio-proxy" Oct 31 03:48:18.264705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.264663 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996"] Oct 31 03:48:18.264865 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.264816 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.283833 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.283562 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-63258b-openshift-default-dockercfg-dsf2l\"" Oct 31 03:48:18.295190 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.295142 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h652q\" (UniqueName: \"kubernetes.io/projected/1ce35023-5f08-4356-8e09-4328952c6040-kube-api-access-h652q\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.295370 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.295279 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-workload-socket\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.295442 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.295406 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce35023-5f08-4356-8e09-4328952c6040-istio-podinfo\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.296284 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.296252 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-workload-certs\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.296478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.296320 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-credential-socket\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.296478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.296445 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-istio-data\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.296579 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.296531 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-istio-envoy\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.296579 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.296572 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce35023-5f08-4356-8e09-4328952c6040-istio-token\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.296771 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.296742 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce35023-5f08-4356-8e09-4328952c6040-istiod-ca-cert\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.360849 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.354246 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:18.473755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.460422 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h652q\" (UniqueName: \"kubernetes.io/projected/1ce35023-5f08-4356-8e09-4328952c6040-kube-api-access-h652q\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.473755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.460549 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-workload-socket\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.473755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.460609 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce35023-5f08-4356-8e09-4328952c6040-istio-podinfo\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.473755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.460710 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-workload-certs\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.473755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.460755 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-credential-socket\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.473755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.460804 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-istio-data\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.473755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.460866 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-istio-envoy\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.473755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.460904 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce35023-5f08-4356-8e09-4328952c6040-istio-token\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.473755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.460950 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce35023-5f08-4356-8e09-4328952c6040-istiod-ca-cert\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.493714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.484541 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-workload-socket\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.523702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.494154 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce35023-5f08-4356-8e09-4328952c6040-istiod-ca-cert\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.523702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.494468 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-istio-data\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.523702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.494718 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-workload-certs\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.523702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.495001 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-credential-socket\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.534666 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.534596 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-istio-envoy\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.653697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.642304 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce35023-5f08-4356-8e09-4328952c6040-istio-token\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.653697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.642391 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h652q\" (UniqueName: \"kubernetes.io/projected/1ce35023-5f08-4356-8e09-4328952c6040-kube-api-access-h652q\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.657029 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.656909 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce35023-5f08-4356-8e09-4328952c6040-istio-podinfo\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.909401 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.909368 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:18.962801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:18.962729 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:19.229750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:19.227819 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:19.260757 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:48:19.260540 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce35023_5f08_4356_8e09_4328952c6040.slice/crio-198ccfc0959cf08c3a48f73203ee530c01bfe908ad3369b0efe34dbdb0b3b210 WatchSource:0}: Error finding container 198ccfc0959cf08c3a48f73203ee530c01bfe908ad3369b0efe34dbdb0b3b210: Status 404 returned error can't find the container with id 198ccfc0959cf08c3a48f73203ee530c01bfe908ad3369b0efe34dbdb0b3b210 Oct 31 03:48:19.261295 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:19.261267 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996"] Oct 31 03:48:19.264689 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:19.264640 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:48:19.264776 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:19.264757 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:48:19.264825 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:19.264793 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:48:19.291854 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:19.291823 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" event={"ID":"1ce35023-5f08-4356-8e09-4328952c6040","Type":"ContainerStarted","Data":"198ccfc0959cf08c3a48f73203ee530c01bfe908ad3369b0efe34dbdb0b3b210"} Oct 31 03:48:19.937069 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:19.926355 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:20.304951 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:20.304855 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" event={"ID":"1ce35023-5f08-4356-8e09-4328952c6040","Type":"ContainerStarted","Data":"07728346be99c829fa088b863aa13c68d084522ef6b407b64fc2ccdf4ad2211e"} Oct 31 03:48:20.332219 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:20.332152 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" podStartSLOduration=2.33213448 podStartE2EDuration="2.33213448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:20.324797169 +0000 UTC m=+1934.859492139" watchObservedRunningTime="2026-10-31 03:48:20.33213448 +0000 UTC m=+1934.866829435" Oct 31 03:48:20.578906 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:20.578805 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:20.910042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:20.910009 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:20.931082 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:20.931048 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:21.226500 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:21.226396 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:21.313978 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:21.313945 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:21.314997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:21.314972 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:48:22.033068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:22.033027 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:22.268880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:22.268851 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" Oct 31 03:48:22.558491 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:22.558444 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:22.701737 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:22.701660 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:22.875870 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:22.873624 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:23.730310 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:23.729279 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:23.895590 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:23.895551 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:24.680698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:24.680236 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:25.300670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:25.300627 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:25.435750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:25.435710 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:25.575610 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:25.575486 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:25.715173 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:25.715140 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:25.989527 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:25.989484 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:26.376197 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:26.376103 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz"] Oct 31 03:48:26.474697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:26.474398 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:26.609120 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:26.609064 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:26.767987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:26.767943 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:27.479479 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:27.479438 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:28.288802 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:28.288761 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:28.976984 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:28.976943 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:29.700666 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:29.700611 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:30.432609 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:30.432555 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:30.993600 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:30.993560 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:31.376860 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:31.376754 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz" podUID="dfc03805-5508-419e-84a2-e6d8494833f7" containerName="payload-pre-processing" containerID="cri-o://19e11d7521d7ce41daf5566961ab97015c1783ff0d41d4755088f0547f1827a1" gracePeriod=25 Oct 31 03:48:31.675085 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:31.675042 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:32.233568 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:32.233530 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:32.364103 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:32.364042 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:32.504993 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:32.504892 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9"] Oct 31 03:48:32.505207 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:32.505171 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" podUID="b3a29402-5f18-4922-b80d-91be39273cd5" containerName="istio-proxy" containerID="cri-o://a4fced6b2f7e94fddef9d6ee41b6cde856d2684f6a604986b7c62978b73d06ca" gracePeriod=30 Oct 31 03:48:32.562370 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:32.562329 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:33.116716 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:33.115414 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:33.764700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:33.751229 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:33.933630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:33.933592 5490 patch_prober.go:29] interesting pod/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:48:33.934481 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:33.933660 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" podUID="b3a29402-5f18-4922-b80d-91be39273cd5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:48:34.145016 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:34.144934 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:34.684454 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:34.684415 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:35.164959 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:35.164918 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz" podUID="dfc03805-5508-419e-84a2-e6d8494833f7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:48:35.379206 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:35.379166 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:36.067801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:36.067739 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:36.787966 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:36.787927 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:37.866147 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.866118 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:48:37.958523 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.958490 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-credential-socket\") pod \"b3a29402-5f18-4922-b80d-91be39273cd5\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " Oct 31 03:48:37.958723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.958574 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-istio-envoy\") pod \"b3a29402-5f18-4922-b80d-91be39273cd5\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " Oct 31 03:48:37.958794 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.958735 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3a29402-5f18-4922-b80d-91be39273cd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3a29402-5f18-4922-b80d-91be39273cd5-istiod-ca-cert\") pod \"b3a29402-5f18-4922-b80d-91be39273cd5\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " Oct 31 03:48:37.958855 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.958833 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-credential-socket" pod "b3a29402-5f18-4922-b80d-91be39273cd5" (UID: "b3a29402-5f18-4922-b80d-91be39273cd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:37.958915 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.958840 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3a29402-5f18-4922-b80d-91be39273cd5-kube-api-access-4mvjt\" (UniqueName: \"kubernetes.io/projected/b3a29402-5f18-4922-b80d-91be39273cd5-kube-api-access-4mvjt\") pod \"b3a29402-5f18-4922-b80d-91be39273cd5\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " Oct 31 03:48:37.958915 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.958896 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3a29402-5f18-4922-b80d-91be39273cd5-istio-token\" (UniqueName: \"kubernetes.io/projected/b3a29402-5f18-4922-b80d-91be39273cd5-istio-token\") pod \"b3a29402-5f18-4922-b80d-91be39273cd5\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " Oct 31 03:48:37.959025 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.958916 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-workload-certs\") pod \"b3a29402-5f18-4922-b80d-91be39273cd5\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " Oct 31 03:48:37.959168 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.959134 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a29402-5f18-4922-b80d-91be39273cd5-istiod-ca-cert" pod "b3a29402-5f18-4922-b80d-91be39273cd5" (UID: "b3a29402-5f18-4922-b80d-91be39273cd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:37.959292 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.959143 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-workload-socket\") pod \"b3a29402-5f18-4922-b80d-91be39273cd5\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " Oct 31 03:48:37.959292 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.959252 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-workload-certs" pod "b3a29402-5f18-4922-b80d-91be39273cd5" (UID: "b3a29402-5f18-4922-b80d-91be39273cd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:37.959415 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.959311 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-workload-socket" pod "b3a29402-5f18-4922-b80d-91be39273cd5" (UID: "b3a29402-5f18-4922-b80d-91be39273cd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:37.959484 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.959447 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-istio-data\") pod \"b3a29402-5f18-4922-b80d-91be39273cd5\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " Oct 31 03:48:37.959544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.959503 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3a29402-5f18-4922-b80d-91be39273cd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3a29402-5f18-4922-b80d-91be39273cd5-istio-podinfo\") pod \"b3a29402-5f18-4922-b80d-91be39273cd5\" (UID: \"b3a29402-5f18-4922-b80d-91be39273cd5\") " Oct 31 03:48:37.959779 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.959751 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-istio-data" pod "b3a29402-5f18-4922-b80d-91be39273cd5" (UID: "b3a29402-5f18-4922-b80d-91be39273cd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:37.960431 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.960403 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:37.960528 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.960436 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:37.960528 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.960451 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3a29402-5f18-4922-b80d-91be39273cd5-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:37.960528 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.960464 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:37.960528 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.960479 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:37.961323 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.961299 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a29402-5f18-4922-b80d-91be39273cd5-kube-api-access-4mvjt" pod "b3a29402-5f18-4922-b80d-91be39273cd5" (UID: "b3a29402-5f18-4922-b80d-91be39273cd5"). InnerVolumeSpecName "kube-api-access-4mvjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:37.961475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.961453 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a29402-5f18-4922-b80d-91be39273cd5-istio-token" pod "b3a29402-5f18-4922-b80d-91be39273cd5" (UID: "b3a29402-5f18-4922-b80d-91be39273cd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:37.961475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.961465 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-istio-envoy" pod "b3a29402-5f18-4922-b80d-91be39273cd5" (UID: "b3a29402-5f18-4922-b80d-91be39273cd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:48:37.961994 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:37.961965 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3a29402-5f18-4922-b80d-91be39273cd5-istio-podinfo" pod "b3a29402-5f18-4922-b80d-91be39273cd5" (UID: "b3a29402-5f18-4922-b80d-91be39273cd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:48:38.064841 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:38.064754 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3a29402-5f18-4922-b80d-91be39273cd5-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:38.064841 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:38.064785 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3a29402-5f18-4922-b80d-91be39273cd5-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:38.064841 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:38.064796 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mvjt\" (UniqueName: \"kubernetes.io/projected/b3a29402-5f18-4922-b80d-91be39273cd5-kube-api-access-4mvjt\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:38.064841 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:38.064806 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3a29402-5f18-4922-b80d-91be39273cd5-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:38.579103 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:38.579061 5490 generic.go:356] "Generic (PLEG): container finished" podID="b3a29402-5f18-4922-b80d-91be39273cd5" containerID="a4fced6b2f7e94fddef9d6ee41b6cde856d2684f6a604986b7c62978b73d06ca" exitCode=0 Oct 31 03:48:38.579453 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:38.579425 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" event={"ID":"b3a29402-5f18-4922-b80d-91be39273cd5","Type":"ContainerDied","Data":"a4fced6b2f7e94fddef9d6ee41b6cde856d2684f6a604986b7c62978b73d06ca"} Oct 31 03:48:38.579532 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:38.579488 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" event={"ID":"b3a29402-5f18-4922-b80d-91be39273cd5","Type":"ContainerDied","Data":"1ca06930a65996802f3a3a80c57fd78d0738f8a4f2305f92bcf9429982a10f84"} Oct 31 03:48:38.579532 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:38.579516 5490 scope.go:122] "RemoveContainer" containerID="a4fced6b2f7e94fddef9d6ee41b6cde856d2684f6a604986b7c62978b73d06ca" Oct 31 03:48:38.579732 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:38.579670 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9" Oct 31 03:48:38.596721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:38.595276 5490 scope.go:122] "RemoveContainer" containerID="a4fced6b2f7e94fddef9d6ee41b6cde856d2684f6a604986b7c62978b73d06ca" Oct 31 03:48:38.596721 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:48:38.596212 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4fced6b2f7e94fddef9d6ee41b6cde856d2684f6a604986b7c62978b73d06ca\": container with ID starting with a4fced6b2f7e94fddef9d6ee41b6cde856d2684f6a604986b7c62978b73d06ca not found: ID does not exist" containerID="a4fced6b2f7e94fddef9d6ee41b6cde856d2684f6a604986b7c62978b73d06ca" Oct 31 03:48:38.596721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:38.596243 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fced6b2f7e94fddef9d6ee41b6cde856d2684f6a604986b7c62978b73d06ca"} err="failed to get container status \"a4fced6b2f7e94fddef9d6ee41b6cde856d2684f6a604986b7c62978b73d06ca\": rpc error: code = NotFound desc = could not find container \"a4fced6b2f7e94fddef9d6ee41b6cde856d2684f6a604986b7c62978b73d06ca\": container with ID starting with a4fced6b2f7e94fddef9d6ee41b6cde856d2684f6a604986b7c62978b73d06ca not found: ID does not exist" Oct 31 03:48:38.596721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:38.596717 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:38.610066 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:38.610036 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9"] Oct 31 03:48:38.617609 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:38.617580 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-880e04-openshift-default-f55d69847-g77p9"] Oct 31 03:48:38.802918 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:38.802877 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:39.109814 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:39.109524 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:39.578165 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:39.578132 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3a29402-5f18-4922-b80d-91be39273cd5" path="/var/lib/kubelet/pods/b3a29402-5f18-4922-b80d-91be39273cd5/volumes" Oct 31 03:48:39.808726 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:39.808670 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:40.538256 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:40.538214 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:41.128707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:41.106793 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:41.977251 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:41.977215 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:42.115007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:42.114948 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:42.261745 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:42.261635 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:42.661167 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:42.661131 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:42.813701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:42.808402 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:42.949631 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:42.949542 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:43.103246 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:43.101377 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:43.425643 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:43.425603 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:44.022703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:44.021298 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:44.896702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:44.891444 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:45.164745 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:45.164702 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz" podUID="dfc03805-5508-419e-84a2-e6d8494833f7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:48:45.668364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:45.668316 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:46.341556 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:46.341514 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:47.038099 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:47.037814 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:47.134077 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:48:47.134041 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:48:47.607081 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:47.607050 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:48.212042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:48.208284 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:48.904708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:48.904035 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:49.588265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:49.588220 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:49.724318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:49.724281 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:50.046274 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:50.046234 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:50.523015 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:50.522934 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:51.658168 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:51.657892 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:51.743505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:51.743468 5490 generic.go:356] "Generic (PLEG): container finished" podID="dfc03805-5508-419e-84a2-e6d8494833f7" containerID="19e11d7521d7ce41daf5566961ab97015c1783ff0d41d4755088f0547f1827a1" exitCode=0 Oct 31 03:48:51.743651 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:51.743541 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz" event={"ID":"dfc03805-5508-419e-84a2-e6d8494833f7","Type":"ContainerDied","Data":"19e11d7521d7ce41daf5566961ab97015c1783ff0d41d4755088f0547f1827a1"} Oct 31 03:48:51.784422 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:51.784383 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz" Oct 31 03:48:51.977863 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:51.977769 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc03805-5508-419e-84a2-e6d8494833f7-kube-api-access-hpd6f\" (UniqueName: \"kubernetes.io/projected/dfc03805-5508-419e-84a2-e6d8494833f7-kube-api-access-hpd6f\") pod \"dfc03805-5508-419e-84a2-e6d8494833f7\" (UID: \"dfc03805-5508-419e-84a2-e6d8494833f7\") " Oct 31 03:48:51.978039 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:51.978024 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc03805-5508-419e-84a2-e6d8494833f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc03805-5508-419e-84a2-e6d8494833f7-plugins-config-volume\") pod \"dfc03805-5508-419e-84a2-e6d8494833f7\" (UID: \"dfc03805-5508-419e-84a2-e6d8494833f7\") " Oct 31 03:48:51.978375 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:51.978346 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc03805-5508-419e-84a2-e6d8494833f7-plugins-config-volume" pod "dfc03805-5508-419e-84a2-e6d8494833f7" (UID: "dfc03805-5508-419e-84a2-e6d8494833f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:48:51.978835 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:51.978809 5490 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc03805-5508-419e-84a2-e6d8494833f7-plugins-config-volume\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:51.980153 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:51.980123 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc03805-5508-419e-84a2-e6d8494833f7-kube-api-access-hpd6f" pod "dfc03805-5508-419e-84a2-e6d8494833f7" (UID: "dfc03805-5508-419e-84a2-e6d8494833f7"). InnerVolumeSpecName "kube-api-access-hpd6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:48:52.081289 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:52.081248 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpd6f\" (UniqueName: \"kubernetes.io/projected/dfc03805-5508-419e-84a2-e6d8494833f7-kube-api-access-hpd6f\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:48:52.472346 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:52.472305 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:52.965022 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:52.963779 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz" event={"ID":"dfc03805-5508-419e-84a2-e6d8494833f7","Type":"ContainerDied","Data":"099d0749c62fcaf412709fbda93c670af5c2d9676cb3106fa5b80f0de2d486ac"} Oct 31 03:48:52.965868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:52.965823 5490 scope.go:122] "RemoveContainer" containerID="19e11d7521d7ce41daf5566961ab97015c1783ff0d41d4755088f0547f1827a1" Oct 31 03:48:52.966232 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:52.966214 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz" Oct 31 03:48:53.095448 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:53.091923 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz"] Oct 31 03:48:53.112321 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:53.112279 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:53.164955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:53.164904 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-73e222-65df574cbc-gghkz"] Oct 31 03:48:53.239228 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:53.239128 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:53.509780 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:53.509667 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:53.588543 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:53.588513 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc03805-5508-419e-84a2-e6d8494833f7" path="/var/lib/kubelet/pods/dfc03805-5508-419e-84a2-e6d8494833f7/volumes" Oct 31 03:48:53.862776 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:53.862584 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:54.494292 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:54.494256 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:55.531711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:55.531594 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:56.223282 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:56.223242 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:56.954697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:56.953712 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:57.704699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:57.701303 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:58.157063 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:58.157026 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:58.547925 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:58.543534 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:59.403820 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:59.398404 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:48:59.913286 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:48:59.913251 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:00.084427 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:00.084386 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:00.863809 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:00.863768 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:01.534826 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:01.529295 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:02.326778 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:02.320894 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:03.188668 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:03.188611 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:04.233960 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:04.233927 5490 generic.go:356] "Generic (PLEG): container finished" podID="015afcb4-d156-4163-a342-f84471c9dd51" containerID="86094c517a636ad0fb95d0395d8f04eaf0b71ce63609059fa53c252f4c2da1e7" exitCode=0 Oct 31 03:49:04.234461 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:04.234006 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" event={"ID":"015afcb4-d156-4163-a342-f84471c9dd51","Type":"ContainerDied","Data":"86094c517a636ad0fb95d0395d8f04eaf0b71ce63609059fa53c252f4c2da1e7"} Oct 31 03:49:04.234718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:04.234661 5490 scope.go:122] "RemoveContainer" containerID="86094c517a636ad0fb95d0395d8f04eaf0b71ce63609059fa53c252f4c2da1e7" Oct 31 03:49:04.236998 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:04.236959 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:49:04.237104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:04.237039 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:49:04.237104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:04.237081 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:49:04.284407 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:04.284371 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:05.013727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:05.013178 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:05.246431 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:05.246397 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" event={"ID":"015afcb4-d156-4163-a342-f84471c9dd51","Type":"ContainerStarted","Data":"b047fbd7d43f633432b3e6284e0e4cd0557b8e084c0bb6dabeca99dade16a78b"} Oct 31 03:49:05.825707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:05.821301 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:05.825707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:05.825193 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:49:05.832566 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:05.831803 5490 patch_prober.go:29] interesting pod/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 31 03:49:05.832566 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:05.831878 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" podUID="015afcb4-d156-4163-a342-f84471c9dd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 31 03:49:06.500701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:06.497702 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:06.820211 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:06.820125 5490 patch_prober.go:29] interesting pod/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 31 03:49:06.820390 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:06.820233 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" podUID="015afcb4-d156-4163-a342-f84471c9dd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 31 03:49:07.320981 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:07.320688 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:07.488076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:07.488043 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t"] Oct 31 03:49:07.488279 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:07.488258 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" podUID="5907e061-56b9-48e5-b9a8-9d33fe49e5a1" containerName="istio-proxy" containerID="cri-o://40d436983fdb913d5e45a8e8fbe4af9c11ee71a8c3b547a04e6f3bde6a64fb95" gracePeriod=30 Oct 31 03:49:07.819665 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:07.819626 5490 patch_prober.go:29] interesting pod/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 31 03:49:07.820181 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:07.819717 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" podUID="015afcb4-d156-4163-a342-f84471c9dd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 31 03:49:08.196785 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:08.196735 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:08.475146 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:08.475059 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:08.820017 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:08.819926 5490 patch_prober.go:29] interesting pod/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 31 03:49:08.820575 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:08.820537 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" podUID="015afcb4-d156-4163-a342-f84471c9dd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 31 03:49:08.988461 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:08.988427 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:09.816403 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:09.816374 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:49:09.819903 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:09.819871 5490 patch_prober.go:29] interesting pod/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 31 03:49:09.820051 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:09.819924 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" podUID="015afcb4-d156-4163-a342-f84471c9dd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 31 03:49:09.858961 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:09.858917 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:10.025604 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:10.025567 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:10.819633 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:10.819599 5490 patch_prober.go:29] interesting pod/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Oct 31 03:49:10.819867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:10.819662 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" podUID="015afcb4-d156-4163-a342-f84471c9dd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Oct 31 03:49:10.934805 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:10.934767 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:11.592661 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:11.592622 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:11.824887 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:11.824858 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:49:11.826180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:11.826156 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:49:12.195357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.195319 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:12.788815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.788789 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:49:12.827014 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.826969 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-credential-socket\") pod \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " Oct 31 03:49:12.827210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.827075 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-workload-certs\") pod \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " Oct 31 03:49:12.827210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.827176 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-workload-socket\") pod \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " Oct 31 03:49:12.827330 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.827210 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-envoy\") pod \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " Oct 31 03:49:12.827330 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.827253 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-kube-api-access-pwlpp\" (UniqueName: \"kubernetes.io/projected/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-kube-api-access-pwlpp\") pod \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " Oct 31 03:49:12.827434 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.827329 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istiod-ca-cert\") pod \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " Oct 31 03:49:12.827434 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.827372 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-token\" (UniqueName: \"kubernetes.io/projected/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-token\") pod \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " Oct 31 03:49:12.827434 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.827423 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-podinfo\") pod \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " Oct 31 03:49:12.830531 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.827475 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-data\") pod \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\" (UID: \"5907e061-56b9-48e5-b9a8-9d33fe49e5a1\") " Oct 31 03:49:12.830531 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.828098 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-workload-socket" pod "5907e061-56b9-48e5-b9a8-9d33fe49e5a1" (UID: "5907e061-56b9-48e5-b9a8-9d33fe49e5a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:12.830531 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.828224 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istiod-ca-cert" pod "5907e061-56b9-48e5-b9a8-9d33fe49e5a1" (UID: "5907e061-56b9-48e5-b9a8-9d33fe49e5a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:49:12.830531 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.828488 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:12.830531 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.828511 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:12.830531 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.828892 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-workload-certs" pod "5907e061-56b9-48e5-b9a8-9d33fe49e5a1" (UID: "5907e061-56b9-48e5-b9a8-9d33fe49e5a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:12.830531 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.828946 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-data" pod "5907e061-56b9-48e5-b9a8-9d33fe49e5a1" (UID: "5907e061-56b9-48e5-b9a8-9d33fe49e5a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:12.830531 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.828931 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-credential-socket" pod "5907e061-56b9-48e5-b9a8-9d33fe49e5a1" (UID: "5907e061-56b9-48e5-b9a8-9d33fe49e5a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:12.830978 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.830646 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-podinfo" pod "5907e061-56b9-48e5-b9a8-9d33fe49e5a1" (UID: "5907e061-56b9-48e5-b9a8-9d33fe49e5a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:49:12.830978 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.830670 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-envoy" pod "5907e061-56b9-48e5-b9a8-9d33fe49e5a1" (UID: "5907e061-56b9-48e5-b9a8-9d33fe49e5a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:12.830978 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.830819 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-token" pod "5907e061-56b9-48e5-b9a8-9d33fe49e5a1" (UID: "5907e061-56b9-48e5-b9a8-9d33fe49e5a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:49:12.831524 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.831492 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-kube-api-access-pwlpp" pod "5907e061-56b9-48e5-b9a8-9d33fe49e5a1" (UID: "5907e061-56b9-48e5-b9a8-9d33fe49e5a1"). InnerVolumeSpecName "kube-api-access-pwlpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:49:12.930932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.930893 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:12.930932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.930923 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:12.930932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.930932 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:12.931259 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.930944 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwlpp\" (UniqueName: \"kubernetes.io/projected/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-kube-api-access-pwlpp\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:12.931259 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.930959 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:12.931259 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.930970 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:12.931259 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:12.930979 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5907e061-56b9-48e5-b9a8-9d33fe49e5a1-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:13.006557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:13.006423 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:13.330560 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:13.330475 5490 generic.go:356] "Generic (PLEG): container finished" podID="5907e061-56b9-48e5-b9a8-9d33fe49e5a1" containerID="40d436983fdb913d5e45a8e8fbe4af9c11ee71a8c3b547a04e6f3bde6a64fb95" exitCode=0 Oct 31 03:49:13.330560 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:13.330538 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" Oct 31 03:49:13.331739 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:13.330561 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" event={"ID":"5907e061-56b9-48e5-b9a8-9d33fe49e5a1","Type":"ContainerDied","Data":"40d436983fdb913d5e45a8e8fbe4af9c11ee71a8c3b547a04e6f3bde6a64fb95"} Oct 31 03:49:13.331739 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:13.330626 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t" event={"ID":"5907e061-56b9-48e5-b9a8-9d33fe49e5a1","Type":"ContainerDied","Data":"c65b9cfab3a5c79e9f79bf40fec560834ede39aacf77b2438c1fdc56a485f9b1"} Oct 31 03:49:13.331739 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:13.330640 5490 scope.go:122] "RemoveContainer" containerID="40d436983fdb913d5e45a8e8fbe4af9c11ee71a8c3b547a04e6f3bde6a64fb95" Oct 31 03:49:13.344465 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:13.344443 5490 scope.go:122] "RemoveContainer" containerID="6ee4efa6d5ab3bb9c86518bbfda5899d81cfff452cbce8623d5b21135c5f28cf" Oct 31 03:49:13.357273 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:13.357240 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t"] Oct 31 03:49:13.357764 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:13.357732 5490 scope.go:122] "RemoveContainer" containerID="40d436983fdb913d5e45a8e8fbe4af9c11ee71a8c3b547a04e6f3bde6a64fb95" Oct 31 03:49:13.358182 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:49:13.358157 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40d436983fdb913d5e45a8e8fbe4af9c11ee71a8c3b547a04e6f3bde6a64fb95\": container with ID starting with 40d436983fdb913d5e45a8e8fbe4af9c11ee71a8c3b547a04e6f3bde6a64fb95 not found: ID does not exist" containerID="40d436983fdb913d5e45a8e8fbe4af9c11ee71a8c3b547a04e6f3bde6a64fb95" Oct 31 03:49:13.358311 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:13.358192 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40d436983fdb913d5e45a8e8fbe4af9c11ee71a8c3b547a04e6f3bde6a64fb95"} err="failed to get container status \"40d436983fdb913d5e45a8e8fbe4af9c11ee71a8c3b547a04e6f3bde6a64fb95\": rpc error: code = NotFound desc = could not find container \"40d436983fdb913d5e45a8e8fbe4af9c11ee71a8c3b547a04e6f3bde6a64fb95\": container with ID starting with 40d436983fdb913d5e45a8e8fbe4af9c11ee71a8c3b547a04e6f3bde6a64fb95 not found: ID does not exist" Oct 31 03:49:13.358311 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:13.358212 5490 scope.go:122] "RemoveContainer" containerID="6ee4efa6d5ab3bb9c86518bbfda5899d81cfff452cbce8623d5b21135c5f28cf" Oct 31 03:49:13.359082 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:49:13.358522 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee4efa6d5ab3bb9c86518bbfda5899d81cfff452cbce8623d5b21135c5f28cf\": container with ID starting with 6ee4efa6d5ab3bb9c86518bbfda5899d81cfff452cbce8623d5b21135c5f28cf not found: ID does not exist" containerID="6ee4efa6d5ab3bb9c86518bbfda5899d81cfff452cbce8623d5b21135c5f28cf" Oct 31 03:49:13.359082 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:13.358551 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee4efa6d5ab3bb9c86518bbfda5899d81cfff452cbce8623d5b21135c5f28cf"} err="failed to get container status \"6ee4efa6d5ab3bb9c86518bbfda5899d81cfff452cbce8623d5b21135c5f28cf\": rpc error: code = NotFound desc = could not find container \"6ee4efa6d5ab3bb9c86518bbfda5899d81cfff452cbce8623d5b21135c5f28cf\": container with ID starting with 6ee4efa6d5ab3bb9c86518bbfda5899d81cfff452cbce8623d5b21135c5f28cf not found: ID does not exist" Oct 31 03:49:13.361950 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:13.361926 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-xqx7t"] Oct 31 03:49:13.577976 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:13.577942 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5907e061-56b9-48e5-b9a8-9d33fe49e5a1" path="/var/lib/kubelet/pods/5907e061-56b9-48e5-b9a8-9d33fe49e5a1/volumes" Oct 31 03:49:13.581245 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:13.581180 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:14.198945 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:14.198860 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:14.802496 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:14.802452 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:15.502377 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:15.501985 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:15.950572 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:15.950523 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:16.260419 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:16.260335 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:17.041730 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:17.041670 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:17.132272 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:49:17.132232 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:49:17.684864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:17.684822 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:17.848418 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:17.848380 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:18.364624 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:18.364576 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:19.048299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:19.048246 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:19.904697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:19.903649 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:20.591254 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:20.591215 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:21.378142 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:21.378104 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:22.066523 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:22.066475 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:22.811693 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:22.811631 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:23.473315 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:23.473278 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:23.791251 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:23.791163 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:24.215361 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:24.215274 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:24.901609 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:24.901570 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:25.643636 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:25.643589 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:26.288344 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:26.288307 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:26.894981 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:26.894942 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:27.728703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:27.726876 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:28.435319 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:28.422958 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:29.063649 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:29.063608 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:29.696116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:29.696079 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:30.016183 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:30.016081 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:30.468074 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:30.468035 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:31.159341 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:31.159302 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:32.005981 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:32.005939 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:32.774094 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:32.774045 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:33.421641 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:33.421591 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:34.083048 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:34.083013 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:34.210082 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:34.210038 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:34.797836 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:34.797795 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:35.707604 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:35.707565 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:36.384983 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:36.384939 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:37.172895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:37.172857 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:37.870855 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:37.870810 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:38.563865 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:38.563827 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:39.223446 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:39.223403 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:39.897711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:39.896744 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:40.526192 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:40.526138 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:41.271756 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:41.271670 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:42.162102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:42.162067 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:42.718544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:42.718507 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s"] Oct 31 03:49:42.719064 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:42.718791 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" podUID="09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" containerName="istio-proxy" containerID="cri-o://2d93e43a037cd18f219826a1027f234e43eb8c6e6b81cf3c4cbfdbc0fdf573fe" gracePeriod=30 Oct 31 03:49:42.723614 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:42.723580 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664"] Oct 31 03:49:42.723870 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:42.723840 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" podUID="a32b3b93-e0e3-4a92-9995-5d69dde3d432" containerName="istio-proxy" containerID="cri-o://368757071fa804f4f8be23437f5661754eb6d16009155bae7e25b457aae1dc90" gracePeriod=30 Oct 31 03:49:43.038099 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:43.038011 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:43.562625 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:43.562589 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:49:43.562803 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:43.562667 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" podUID="09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:49:43.807887 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:43.803977 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:44.429367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:44.429325 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:45.099830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:45.099793 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:45.822204 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:45.822168 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:46.488852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:46.488806 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:47.115088 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:47.115049 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:47.159903 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:49:47.159834 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:49:47.703040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:47.703002 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:48.033346 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.033310 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:49:48.074829 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.074791 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-workload-socket\") pod \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " Oct 31 03:49:48.074986 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.074880 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-podinfo\") pod \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " Oct 31 03:49:48.074986 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.074966 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-kube-api-access-bz6wf\" (UniqueName: \"kubernetes.io/projected/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-kube-api-access-bz6wf\") pod \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " Oct 31 03:49:48.075101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.075057 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istiod-ca-cert\") pod \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " Oct 31 03:49:48.075101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.075090 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-data\") pod \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " Oct 31 03:49:48.075203 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.075099 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-workload-socket" pod "09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" (UID: "09be0ab9-60dd-4f14-ba3f-75aeedb1a62f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:48.075203 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.075176 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-credential-socket\") pod \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " Oct 31 03:49:48.075300 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.075216 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-envoy\") pod \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " Oct 31 03:49:48.075355 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.075300 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-token\" (UniqueName: \"kubernetes.io/projected/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-token\") pod \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " Oct 31 03:49:48.075485 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.075462 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-workload-certs\") pod \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\" (UID: \"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f\") " Oct 31 03:49:48.076394 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.076368 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:48.076686 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.076628 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-workload-certs" pod "09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" (UID: "09be0ab9-60dd-4f14-ba3f-75aeedb1a62f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:48.077140 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.076947 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istiod-ca-cert" pod "09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" (UID: "09be0ab9-60dd-4f14-ba3f-75aeedb1a62f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:49:48.077292 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.077210 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-data" pod "09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" (UID: "09be0ab9-60dd-4f14-ba3f-75aeedb1a62f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:48.077444 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.077399 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-credential-socket" pod "09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" (UID: "09be0ab9-60dd-4f14-ba3f-75aeedb1a62f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:48.082619 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.082590 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-kube-api-access-bz6wf" pod "09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" (UID: "09be0ab9-60dd-4f14-ba3f-75aeedb1a62f"). InnerVolumeSpecName "kube-api-access-bz6wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:49:48.083147 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.083110 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-envoy" pod "09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" (UID: "09be0ab9-60dd-4f14-ba3f-75aeedb1a62f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:48.083256 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.083156 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-token" pod "09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" (UID: "09be0ab9-60dd-4f14-ba3f-75aeedb1a62f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:49:48.084864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.084842 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-podinfo" pod "09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" (UID: "09be0ab9-60dd-4f14-ba3f-75aeedb1a62f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:49:48.100606 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.100528 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:49:48.177198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.177150 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-credential-socket\") pod \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " Oct 31 03:49:48.177363 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.177277 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-data\") pod \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " Oct 31 03:49:48.177363 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.177320 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-token\" (UniqueName: \"kubernetes.io/projected/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-token\") pod \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " Oct 31 03:49:48.177494 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.177394 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-envoy\") pod \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " Oct 31 03:49:48.177494 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.177427 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istiod-ca-cert\") pod \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " Oct 31 03:49:48.177494 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.177476 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-workload-socket\") pod \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " Oct 31 03:49:48.177615 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.177509 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-workload-certs\") pod \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " Oct 31 03:49:48.178276 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.177712 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a32b3b93-e0e3-4a92-9995-5d69dde3d432-kube-api-access-56k62\" (UniqueName: \"kubernetes.io/projected/a32b3b93-e0e3-4a92-9995-5d69dde3d432-kube-api-access-56k62\") pod \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " Oct 31 03:49:48.178276 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.177928 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-podinfo\") pod \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\" (UID: \"a32b3b93-e0e3-4a92-9995-5d69dde3d432\") " Oct 31 03:49:48.178276 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.178096 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-data" pod "a32b3b93-e0e3-4a92-9995-5d69dde3d432" (UID: "a32b3b93-e0e3-4a92-9995-5d69dde3d432"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:48.178276 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.178113 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-credential-socket" pod "a32b3b93-e0e3-4a92-9995-5d69dde3d432" (UID: "a32b3b93-e0e3-4a92-9995-5d69dde3d432"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:48.179278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.178731 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:48.179278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.178793 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:48.179278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.178811 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:48.179278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.178826 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:48.179278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.178839 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:48.179278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.178824 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-workload-socket" pod "a32b3b93-e0e3-4a92-9995-5d69dde3d432" (UID: "a32b3b93-e0e3-4a92-9995-5d69dde3d432"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:48.179278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.178854 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:48.179278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.178895 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:48.179278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.178918 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:48.179278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.178934 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:48.179278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.178949 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz6wf\" (UniqueName: \"kubernetes.io/projected/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f-kube-api-access-bz6wf\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:48.179278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.178891 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-workload-certs" pod "a32b3b93-e0e3-4a92-9995-5d69dde3d432" (UID: "a32b3b93-e0e3-4a92-9995-5d69dde3d432"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:48.179278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.179231 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istiod-ca-cert" pod "a32b3b93-e0e3-4a92-9995-5d69dde3d432" (UID: "a32b3b93-e0e3-4a92-9995-5d69dde3d432"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:49:48.180486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.180464 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-token" pod "a32b3b93-e0e3-4a92-9995-5d69dde3d432" (UID: "a32b3b93-e0e3-4a92-9995-5d69dde3d432"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:49:48.181506 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.181485 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a32b3b93-e0e3-4a92-9995-5d69dde3d432-kube-api-access-56k62" pod "a32b3b93-e0e3-4a92-9995-5d69dde3d432" (UID: "a32b3b93-e0e3-4a92-9995-5d69dde3d432"). InnerVolumeSpecName "kube-api-access-56k62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:49:48.181659 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.181635 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-envoy" pod "a32b3b93-e0e3-4a92-9995-5d69dde3d432" (UID: "a32b3b93-e0e3-4a92-9995-5d69dde3d432"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:49:48.182365 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.181814 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-podinfo" pod "a32b3b93-e0e3-4a92-9995-5d69dde3d432" (UID: "a32b3b93-e0e3-4a92-9995-5d69dde3d432"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:49:48.281209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.281122 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:48.281209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.281154 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:48.281209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.281164 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:48.281209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.281173 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:48.281209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.281182 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a32b3b93-e0e3-4a92-9995-5d69dde3d432-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:48.281209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.281192 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56k62\" (UniqueName: \"kubernetes.io/projected/a32b3b93-e0e3-4a92-9995-5d69dde3d432-kube-api-access-56k62\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:48.281209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.281201 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a32b3b93-e0e3-4a92-9995-5d69dde3d432-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:49:48.406902 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.406603 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:48.695958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.695911 5490 generic.go:356] "Generic (PLEG): container finished" podID="a32b3b93-e0e3-4a92-9995-5d69dde3d432" containerID="368757071fa804f4f8be23437f5661754eb6d16009155bae7e25b457aae1dc90" exitCode=0 Oct 31 03:49:48.696158 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.695972 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" Oct 31 03:49:48.696158 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.696112 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" event={"ID":"a32b3b93-e0e3-4a92-9995-5d69dde3d432","Type":"ContainerDied","Data":"368757071fa804f4f8be23437f5661754eb6d16009155bae7e25b457aae1dc90"} Oct 31 03:49:48.696269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.696166 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664" event={"ID":"a32b3b93-e0e3-4a92-9995-5d69dde3d432","Type":"ContainerDied","Data":"893e31b4258457d439e1503fa705ee5fa5e26281920eecf51375f455e9dabbea"} Oct 31 03:49:48.696269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.696194 5490 scope.go:122] "RemoveContainer" containerID="368757071fa804f4f8be23437f5661754eb6d16009155bae7e25b457aae1dc90" Oct 31 03:49:48.698198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.698175 5490 generic.go:356] "Generic (PLEG): container finished" podID="09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" containerID="2d93e43a037cd18f219826a1027f234e43eb8c6e6b81cf3c4cbfdbc0fdf573fe" exitCode=0 Oct 31 03:49:48.698307 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.698253 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" Oct 31 03:49:48.698307 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.698297 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" event={"ID":"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f","Type":"ContainerDied","Data":"2d93e43a037cd18f219826a1027f234e43eb8c6e6b81cf3c4cbfdbc0fdf573fe"} Oct 31 03:49:48.698390 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.698335 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s" event={"ID":"09be0ab9-60dd-4f14-ba3f-75aeedb1a62f","Type":"ContainerDied","Data":"bb652916a26ab7283f797531401bc4c1ff940163c1b7356dc8dc03c0cbe8a1ff"} Oct 31 03:49:48.721189 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.721160 5490 scope.go:122] "RemoveContainer" containerID="368757071fa804f4f8be23437f5661754eb6d16009155bae7e25b457aae1dc90" Oct 31 03:49:48.722437 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:49:48.722397 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368757071fa804f4f8be23437f5661754eb6d16009155bae7e25b457aae1dc90\": container with ID starting with 368757071fa804f4f8be23437f5661754eb6d16009155bae7e25b457aae1dc90 not found: ID does not exist" containerID="368757071fa804f4f8be23437f5661754eb6d16009155bae7e25b457aae1dc90" Oct 31 03:49:48.722543 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.722444 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368757071fa804f4f8be23437f5661754eb6d16009155bae7e25b457aae1dc90"} err="failed to get container status \"368757071fa804f4f8be23437f5661754eb6d16009155bae7e25b457aae1dc90\": rpc error: code = NotFound desc = could not find container \"368757071fa804f4f8be23437f5661754eb6d16009155bae7e25b457aae1dc90\": container with ID starting with 368757071fa804f4f8be23437f5661754eb6d16009155bae7e25b457aae1dc90 not found: ID does not exist" Oct 31 03:49:48.722543 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.722471 5490 scope.go:122] "RemoveContainer" containerID="2d93e43a037cd18f219826a1027f234e43eb8c6e6b81cf3c4cbfdbc0fdf573fe" Oct 31 03:49:48.731245 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.731213 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664"] Oct 31 03:49:48.735691 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.735648 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-zl664"] Oct 31 03:49:48.741747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.741725 5490 scope.go:122] "RemoveContainer" containerID="2d93e43a037cd18f219826a1027f234e43eb8c6e6b81cf3c4cbfdbc0fdf573fe" Oct 31 03:49:48.742119 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:49:48.742095 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d93e43a037cd18f219826a1027f234e43eb8c6e6b81cf3c4cbfdbc0fdf573fe\": container with ID starting with 2d93e43a037cd18f219826a1027f234e43eb8c6e6b81cf3c4cbfdbc0fdf573fe not found: ID does not exist" containerID="2d93e43a037cd18f219826a1027f234e43eb8c6e6b81cf3c4cbfdbc0fdf573fe" Oct 31 03:49:48.742170 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.742127 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d93e43a037cd18f219826a1027f234e43eb8c6e6b81cf3c4cbfdbc0fdf573fe"} err="failed to get container status \"2d93e43a037cd18f219826a1027f234e43eb8c6e6b81cf3c4cbfdbc0fdf573fe\": rpc error: code = NotFound desc = could not find container \"2d93e43a037cd18f219826a1027f234e43eb8c6e6b81cf3c4cbfdbc0fdf573fe\": container with ID starting with 2d93e43a037cd18f219826a1027f234e43eb8c6e6b81cf3c4cbfdbc0fdf573fe not found: ID does not exist" Oct 31 03:49:48.744816 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.744784 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s"] Oct 31 03:49:48.748885 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:48.748865 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-hbs7s"] Oct 31 03:49:49.204025 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:49.203989 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:49.578437 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:49.578356 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" path="/var/lib/kubelet/pods/09be0ab9-60dd-4f14-ba3f-75aeedb1a62f/volumes" Oct 31 03:49:49.579021 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:49.579003 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a32b3b93-e0e3-4a92-9995-5d69dde3d432" path="/var/lib/kubelet/pods/a32b3b93-e0e3-4a92-9995-5d69dde3d432/volumes" Oct 31 03:49:49.930872 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:49.930835 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:50.818343 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:50.818303 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:52.164397 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:52.164287 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:52.432083 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:52.431973 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:49:53.816015 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:53.815973 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:54.686086 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:54.686037 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:55.661807 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:55.661726 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:56.545700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:56.540317 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:57.432158 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:57.432118 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:58.309789 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:58.309751 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:59.033610 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:59.033506 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:49:59.799539 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:49:59.799496 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:00.474279 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:00.474242 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:01.124669 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:01.124569 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:01.995184 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:01.993814 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:02.575173 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:02.575136 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:03.180247 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:03.180206 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:03.935650 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:03.935612 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:04.093378 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:04.093346 5490 generic.go:356] "Generic (PLEG): container finished" podID="1ce35023-5f08-4356-8e09-4328952c6040" containerID="07728346be99c829fa088b863aa13c68d084522ef6b407b64fc2ccdf4ad2211e" exitCode=0 Oct 31 03:50:04.093574 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:04.093425 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" event={"ID":"1ce35023-5f08-4356-8e09-4328952c6040","Type":"ContainerDied","Data":"07728346be99c829fa088b863aa13c68d084522ef6b407b64fc2ccdf4ad2211e"} Oct 31 03:50:04.093875 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:04.093858 5490 scope.go:122] "RemoveContainer" containerID="07728346be99c829fa088b863aa13c68d084522ef6b407b64fc2ccdf4ad2211e" Oct 31 03:50:04.096030 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:04.095975 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:50:04.096142 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:04.096074 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:50:04.096142 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:04.096110 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:50:04.603516 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:04.603421 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:05.107855 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:05.107817 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" event={"ID":"1ce35023-5f08-4356-8e09-4328952c6040","Type":"ContainerStarted","Data":"de079737f212ac8275e31adf6ab22910b8efefe3c4d6bd57862bdc43663ad8b4"} Oct 31 03:50:05.654041 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:05.653987 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:05.910403 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:05.910316 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:50:05.943707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:05.936293 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:50:05.944031 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:05.943992 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" podUID="1ce35023-5f08-4356-8e09-4328952c6040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 31 03:50:06.313906 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:06.313809 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:50:06.399928 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:06.399868 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:06.910282 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:06.910248 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:50:06.910803 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:06.910323 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" podUID="1ce35023-5f08-4356-8e09-4328952c6040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 31 03:50:07.102761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:07.102715 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:07.822943 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:07.822906 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:07.910320 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:07.910284 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:50:07.910852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:07.910360 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" podUID="1ce35023-5f08-4356-8e09-4328952c6040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 31 03:50:08.680575 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:08.680528 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:08.909872 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:08.909826 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:50:08.910054 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:08.909891 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" podUID="1ce35023-5f08-4356-8e09-4328952c6040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 31 03:50:09.617374 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:09.617320 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:09.909811 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:09.909774 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 31 03:50:09.910008 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:09.909841 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" podUID="1ce35023-5f08-4356-8e09-4328952c6040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 31 03:50:10.329606 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:10.329504 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:10.899089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:10.899047 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:10.923698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:10.922231 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:50:10.929728 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:10.924095 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:50:11.027419 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:11.027381 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:11.593686 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:11.590818 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:12.242407 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:12.242357 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:12.910541 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:12.910498 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:14.206827 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:14.206779 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:14.869981 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:14.869883 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:15.469663 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:15.469150 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:16.172452 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:16.172410 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:16.782555 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:16.780086 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:17.133996 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:50:17.133895 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:50:17.384174 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:17.384093 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:17.831516 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:17.831469 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:18.071692 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:18.071644 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:18.676084 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:18.676022 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:19.277695 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:19.277640 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:19.913991 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:19.913097 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:20.547537 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:20.547481 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:21.220379 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:21.220343 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:21.823860 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:21.823819 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:22.613740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:22.613307 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:23.316456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:23.316407 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:23.945249 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:23.945206 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:24.590558 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:24.590514 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:25.196150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:25.196113 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:25.803250 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:25.803200 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:26.626106 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:26.626061 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:27.284521 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:27.284478 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:27.883636 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:27.883597 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:28.547034 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:28.546991 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:29.187949 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:29.187913 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:29.882931 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:29.882887 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:30.865150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:30.865105 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:31.644319 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:31.644277 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:31.810497 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:31.810461 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:32.485267 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:32.485223 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:33.227259 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:33.227221 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:33.961463 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:33.961402 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:34.881779 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:34.881742 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:35.493377 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:35.493335 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:36.243330 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:36.243290 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:37.194288 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.194239 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:37.746364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.746329 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h"] Oct 31 03:50:37.747455 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.747425 5490 cpu_manager.go:414] "containerMap: removing container" podUID="5907e061-56b9-48e5-b9a8-9d33fe49e5a1" containerName="istio-proxy" Oct 31 03:50:37.747455 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.747453 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5907e061-56b9-48e5-b9a8-9d33fe49e5a1" containerName="istio-proxy" Oct 31 03:50:37.747664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.747511 5490 cpu_manager.go:414] "containerMap: removing container" podUID="b3a29402-5f18-4922-b80d-91be39273cd5" containerName="istio-proxy" Oct 31 03:50:37.747664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.747522 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3a29402-5f18-4922-b80d-91be39273cd5" containerName="istio-proxy" Oct 31 03:50:37.747664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.747544 5490 cpu_manager.go:414] "containerMap: removing container" podUID="5907e061-56b9-48e5-b9a8-9d33fe49e5a1" containerName="istio-proxy" Oct 31 03:50:37.747664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.747550 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5907e061-56b9-48e5-b9a8-9d33fe49e5a1" containerName="istio-proxy" Oct 31 03:50:37.747664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.747566 5490 cpu_manager.go:414] "containerMap: removing container" podUID="a32b3b93-e0e3-4a92-9995-5d69dde3d432" containerName="istio-proxy" Oct 31 03:50:37.747664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.747570 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a32b3b93-e0e3-4a92-9995-5d69dde3d432" containerName="istio-proxy" Oct 31 03:50:37.747664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.747578 5490 cpu_manager.go:414] "containerMap: removing container" podUID="09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" containerName="istio-proxy" Oct 31 03:50:37.747664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.747583 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" containerName="istio-proxy" Oct 31 03:50:37.747664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.747590 5490 cpu_manager.go:414] "containerMap: removing container" podUID="dfc03805-5508-419e-84a2-e6d8494833f7" containerName="payload-pre-processing" Oct 31 03:50:37.747664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.747594 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc03805-5508-419e-84a2-e6d8494833f7" containerName="payload-pre-processing" Oct 31 03:50:37.747999 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.747747 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="a32b3b93-e0e3-4a92-9995-5d69dde3d432" containerName="istio-proxy" Oct 31 03:50:37.747999 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.747759 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="09be0ab9-60dd-4f14-ba3f-75aeedb1a62f" containerName="istio-proxy" Oct 31 03:50:37.747999 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.747767 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc03805-5508-419e-84a2-e6d8494833f7" containerName="payload-pre-processing" Oct 31 03:50:37.747999 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.747778 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="5907e061-56b9-48e5-b9a8-9d33fe49e5a1" containerName="istio-proxy" Oct 31 03:50:37.747999 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.747786 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3a29402-5f18-4922-b80d-91be39273cd5" containerName="istio-proxy" Oct 31 03:50:37.747999 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.747795 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="5907e061-56b9-48e5-b9a8-9d33fe49e5a1" containerName="istio-proxy" Oct 31 03:50:37.752282 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.752260 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:37.762179 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.762151 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-afb8be-openshift-default-dockercfg-gvlck\"" Oct 31 03:50:37.764773 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.764748 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh"] Oct 31 03:50:37.771401 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.771375 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h"] Oct 31 03:50:37.771596 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.771481 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:37.784783 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.784752 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh"] Oct 31 03:50:37.929274 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.929230 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:37.929440 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.929289 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:37.929485 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.929449 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:37.929485 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.929476 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f57cc204-16b4-4735-82c1-7cee96c1704d-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:37.929630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.929605 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:37.930278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.929648 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:37.930278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.929706 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:37.930278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.929764 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:37.930278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.929888 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:37.930278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.929956 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w775l\" (UniqueName: \"kubernetes.io/projected/8317b14b-20f5-4404-8f4c-bf315ec53736-kube-api-access-w775l\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:37.930278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.930016 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:37.930278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.930098 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:37.930278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.930131 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:37.930278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.930164 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:37.930278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.930271 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:37.931407 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.930308 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shjhj\" (UniqueName: \"kubernetes.io/projected/f57cc204-16b4-4735-82c1-7cee96c1704d-kube-api-access-shjhj\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:37.931407 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.930391 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8317b14b-20f5-4404-8f4c-bf315ec53736-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:37.931407 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.930417 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:37.980920 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:37.980880 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:38.032456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.032366 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:38.032456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.032416 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.032459 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.032577 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.032705 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.032763 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w775l\" (UniqueName: \"kubernetes.io/projected/8317b14b-20f5-4404-8f4c-bf315ec53736-kube-api-access-w775l\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.032811 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.032865 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.032900 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.032939 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.032958 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.033024 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.033062 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shjhj\" (UniqueName: \"kubernetes.io/projected/f57cc204-16b4-4735-82c1-7cee96c1704d-kube-api-access-shjhj\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.033076 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.033093 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.033140 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8317b14b-20f5-4404-8f4c-bf315ec53736-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.033171 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.033189 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.033211 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.033255 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.033345 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.033356 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.033383 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.033385 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.033426 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f57cc204-16b4-4735-82c1-7cee96c1704d-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.033580 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:38.034068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.033950 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8317b14b-20f5-4404-8f4c-bf315ec53736-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:38.036571 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.036537 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f57cc204-16b4-4735-82c1-7cee96c1704d-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:38.041088 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.039671 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:38.041088 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.039904 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:38.041088 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.040185 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:38.041088 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.040215 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:38.041892 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.041862 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:38.041974 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.041867 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w775l\" (UniqueName: \"kubernetes.io/projected/8317b14b-20f5-4404-8f4c-bf315ec53736-kube-api-access-w775l\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:38.042167 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.042150 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:38.042213 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.042175 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shjhj\" (UniqueName: \"kubernetes.io/projected/f57cc204-16b4-4735-82c1-7cee96c1704d-kube-api-access-shjhj\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:38.073981 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.073941 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:38.103977 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.103951 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:38.422759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.422730 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh"] Oct 31 03:50:38.423539 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:50:38.423507 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57cc204_16b4_4735_82c1_7cee96c1704d.slice/crio-7a99bdb12b9060c5cf02942ab341f715cc430a9f268d937a3e8c0ce4bb1af3ff WatchSource:0}: Error finding container 7a99bdb12b9060c5cf02942ab341f715cc430a9f268d937a3e8c0ce4bb1af3ff: Status 404 returned error can't find the container with id 7a99bdb12b9060c5cf02942ab341f715cc430a9f268d937a3e8c0ce4bb1af3ff Oct 31 03:50:38.425044 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:50:38.424929 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8317b14b_20f5_4404_8f4c_bf315ec53736.slice/crio-928bb517591b623cdcc2ae0773c070758112ff881c842bc2f4a52c923959e550 WatchSource:0}: Error finding container 928bb517591b623cdcc2ae0773c070758112ff881c842bc2f4a52c923959e550: Status 404 returned error can't find the container with id 928bb517591b623cdcc2ae0773c070758112ff881c842bc2f4a52c923959e550 Oct 31 03:50:38.427550 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.427507 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:50:38.427658 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.427600 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:50:38.427782 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.427657 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:50:38.428173 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.428135 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:50:38.428281 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.428218 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:50:38.428339 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.428309 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:50:38.428810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.428770 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h"] Oct 31 03:50:38.493095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.493046 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" event={"ID":"8317b14b-20f5-4404-8f4c-bf315ec53736","Type":"ContainerStarted","Data":"928bb517591b623cdcc2ae0773c070758112ff881c842bc2f4a52c923959e550"} Oct 31 03:50:38.496520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.496303 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" event={"ID":"f57cc204-16b4-4735-82c1-7cee96c1704d","Type":"ContainerStarted","Data":"7a99bdb12b9060c5cf02942ab341f715cc430a9f268d937a3e8c0ce4bb1af3ff"} Oct 31 03:50:38.699390 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:38.699282 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:39.478702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:39.476459 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:39.620533 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:39.620493 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" event={"ID":"8317b14b-20f5-4404-8f4c-bf315ec53736","Type":"ContainerStarted","Data":"75d33ed8265ccc3c5cd1979a992cbb481d894e7d6715fc035d67fb91b327dccf"} Oct 31 03:50:39.620725 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:39.620600 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" event={"ID":"f57cc204-16b4-4735-82c1-7cee96c1704d","Type":"ContainerStarted","Data":"977556c538ae57240b5b254d28dbffae276a3566ad2bf1217ca04aa05d4bf3b0"} Oct 31 03:50:39.668698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:39.668571 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" podStartSLOduration=2.668550331 podStartE2EDuration="2.668550331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:39.636007479 +0000 UTC m=+2074.170702436" watchObservedRunningTime="2026-10-31 03:50:39.668550331 +0000 UTC m=+2074.203245288" Oct 31 03:50:39.690078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:39.689980 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" podStartSLOduration=2.689960926 podStartE2EDuration="2.689960926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:39.668206247 +0000 UTC m=+2074.202901204" watchObservedRunningTime="2026-10-31 03:50:39.689960926 +0000 UTC m=+2074.224655893" Oct 31 03:50:40.088699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:40.087732 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:40.089430 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:40.089398 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 31 03:50:40.089569 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:40.089460 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" podUID="8317b14b-20f5-4404-8f4c-bf315ec53736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 31 03:50:40.105710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:40.104401 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:40.105891 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:40.105724 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 31 03:50:40.105891 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:40.105782 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" podUID="f57cc204-16b4-4735-82c1-7cee96c1704d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 31 03:50:40.308407 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:40.308374 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:41.079699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:41.078756 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 31 03:50:41.099041 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:41.098987 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" podUID="8317b14b-20f5-4404-8f4c-bf315ec53736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 31 03:50:41.101776 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:41.101713 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:41.104943 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:41.104913 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 31 03:50:41.105047 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:41.104964 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" podUID="f57cc204-16b4-4735-82c1-7cee96c1704d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 31 03:50:42.077176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:42.077132 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 31 03:50:42.077422 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:42.077199 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" podUID="8317b14b-20f5-4404-8f4c-bf315ec53736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 31 03:50:42.105203 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:42.105166 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 31 03:50:42.105716 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:42.105666 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" podUID="f57cc204-16b4-4735-82c1-7cee96c1704d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 31 03:50:42.138707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:42.135176 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:43.075698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:43.075017 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 31 03:50:43.075698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:43.075094 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" podUID="8317b14b-20f5-4404-8f4c-bf315ec53736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 31 03:50:43.093794 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:43.093756 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:43.112548 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:43.112510 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 31 03:50:43.113279 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:43.113243 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" podUID="f57cc204-16b4-4735-82c1-7cee96c1704d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 31 03:50:43.962701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:43.961838 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:44.077144 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:44.077101 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 31 03:50:44.077451 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:44.077418 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" podUID="8317b14b-20f5-4404-8f4c-bf315ec53736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 31 03:50:44.110705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:44.109473 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 31 03:50:44.110705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:44.109543 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" podUID="f57cc204-16b4-4735-82c1-7cee96c1704d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 31 03:50:44.596419 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:44.596376 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:45.074395 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:45.074355 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 31 03:50:45.074577 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:45.074426 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" podUID="8317b14b-20f5-4404-8f4c-bf315ec53736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 31 03:50:45.104979 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:45.104947 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 31 03:50:45.118364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:45.105013 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" podUID="f57cc204-16b4-4735-82c1-7cee96c1704d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 31 03:50:46.565853 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:46.552799 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:46.849955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:46.849870 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:46.880540 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:46.851667 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:46.880540 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:46.853816 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:50:47.104951 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:47.104856 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:50:47.104951 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:47.104931 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" podUID="f57cc204-16b4-4735-82c1-7cee96c1704d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:47.204328 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:47.110109 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:47.204328 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:47.110567 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:47.204328 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:47.111341 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:50:48.221895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:48.221839 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:48.367579 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:48.224450 5490 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-5kbd8 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.11:9443/health\": context deadline exceeded" start-of-body= Oct 31 03:50:48.367579 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:48.224488 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" podUID="8bae2714-9c1b-44db-a6ef-b0b008cf1fa2" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.133.0.11:9443/health\": context deadline exceeded" Oct 31 03:50:48.513128 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:48.513020 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" podUID="55f29950-78b4-4a52-818d-dc235e4631fc" containerName="proxy" probeResult="failure" output="Get \"https://10.133.0.36:8443/ready\": context deadline exceeded" Oct 31 03:50:49.630204 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:49.630156 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:49.630776 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:49.630360 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:50:49.733013 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:49.721047 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:50:50.088879 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:50.088828 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/odh-model-controller-769875db56-6r4mx" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.32:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:50.121843 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:50.112760 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" podUID="85995108-6b7b-4e56-8260-00b5347d6e6a" containerName="postgres" probeResult="failure" output="command timed out" Oct 31 03:50:50.619359 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:50.619277 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:50:51.429869 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:51.429816 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:51.469599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:51.466792 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:53.788856 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:50:53.788805 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:50:54.788939 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:54.788890 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:55.011182 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:54.803297 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:50:55.318753 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:55.318707 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:55.372162 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:55.343365 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:55.372162 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:55.355062 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" podUID="253719d9-0fac-4d41-bfc3-d4d359ba04fe" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.135:8443/health\": context deadline exceeded" Oct 31 03:50:55.503851 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:55.391530 5490 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ttmkb container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:50:55.503851 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:55.391577 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" podUID="400b676a-a3a0-459d-9da6-8d30b2d3b497" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:55.503851 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:55.395234 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" podUID="f64d3e84-440a-4298-9230-ae9e0063c962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 03:50:55.503851 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:55.396066 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" podUID="f64d3e84-440a-4298-9230-ae9e0063c962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded" Oct 31 03:50:55.503851 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:55.396245 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-6r4mx" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.32:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:55.750311 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:55.524770 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Oct 31 03:50:55.750311 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:55.525053 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:50:55.899048 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:55.878474 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:50:55.899048 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:55.880220 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": context deadline exceeded" Oct 31 03:50:56.302188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:55.904848 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:56.517327 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:56.340969 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:56.517327 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:56.342070 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" podUID="775a8940-cb22-4774-a43f-c6e7c5364354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:56.517327 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:56.392581 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-586ddd746f-nv8gm" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:56.517327 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:56.396657 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" podUID="f64d3e84-440a-4298-9230-ae9e0063c962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:56.913612 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:56.526032 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:50:56.913612 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:56.625964 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:56.913612 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:56.628096 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 03:50:56.913612 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:56.802932 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" podUID="e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.101:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:56.913612 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:56.803062 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" podUID="669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.57:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:56.913612 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:56.875078 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:56.913612 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:56.880245 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:56.913612 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:56.885557 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" podUID="80180652-fb4f-4976-956d-25a47344d581" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:50:57.151293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:57.027221 5490 patch_prober.go:29] interesting pod/thanos-querier-b949d9bf6-8b6xc container/kube-rbac-proxy-web namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.133.0.10:9091/-/healthy\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:50:57.151293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:57.027320 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/healthy\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:00.065789 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:00.065737 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:51:00.103557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:00.066603 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:00.313075 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:00.313038 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:00.313295 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:00.313091 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" podUID="b0b3f9b0-9f95-4556-87a2-0d7473db75f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:00.362830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:00.344483 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" podUID="6f4d14ca-ec36-4daa-8971-93c7dd709b48" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.39:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:00.362830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:00.344490 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" podUID="80180652-fb4f-4976-956d-25a47344d581" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:00.362830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:00.345128 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:51:00.362830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:00.346188 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" podUID="b7c0d839-b3a4-463b-a3a8-3733609d4b89" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.134:8443/health\": context deadline exceeded" Oct 31 03:51:00.922723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:56.344219 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:00.922723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:00.381775 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" podUID="efd854cd-40a6-4f9f-bd19-cd5d19512dba" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.24:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:00.922723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:00.458286 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" podUID="da913fbf-0ba8-4607-be4c-5197a22d1a7d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.133:8443/health\": write tcp 10.133.0.2:57222->10.133.0.133:8443: write: broken pipe" Oct 31 03:51:00.922723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:00.484210 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" podUID="253719d9-0fac-4d41-bfc3-d4d359ba04fe" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.135:8443/health\": context deadline exceeded" Oct 31 03:51:00.922723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:00.486161 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.539211715s: [/var/lib/containers/storage/overlay/6ca5d3dc094fd0bd9cfa2606b333ca9db05c965371229c21a2ab0726b6feee44/diff /var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rclxh_9e7f6e11-0382-4897-88e3-1d4283f5bed3/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:51:00.922723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:00.499873 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" podUID="30874c80-e957-4c9a-ab8f-0616de567c3d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:00.922723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:00.501595 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:00.922723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:00.501665 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:00.922723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:56.319190 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:00.922723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:00.506507 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.080276667s: [/var/lib/containers/storage/overlay/d4ff1777ab4b2fd5c4da7d420419184ee8f9946db1ed830ec56007f96640e090/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs_e9f75494-3ba5-460d-a96d-9ab2f4ffa395/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:51:00.922723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:56.340955 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:00.922723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:00.635467 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:00.922723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:00.067013 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.11997036s: [/var/lib/containers/storage/overlay/95662cc85fdec0b3d5e058b73867d857ca9b05b858ee8c8c8a6e76e10d1d9964/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd_bae51678-b17d-4a9c-98da-77d3ecf0d6ed/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.250301 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.250345 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" podUID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.599139 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.599200 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" podUID="dacc99d6-e9da-46e0-841d-14c5f54dd8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.601983 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.655720889s: [/var/lib/containers/storage/overlay/520654143d43f8c05469e9cf12f580ed3c3fdb3237a6a4039f4fe85621c02f02/diff /var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-gzjbq_c9d776b1-fabe-4396-adb1-030d1edad90c/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:50:56.482827 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd"] Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.698988 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" podUID="42e53881-926d-482d-9557-a86d3ae3f9d9" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.98:50052\" within 1s: context deadline exceeded" Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.716218 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.724146 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" podUID="5123bf1e-0ad6-4406-86b0-2b9a9e025a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.717052 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" podUID="e9f75494-3ba5-460d-a96d-9ab2f4ffa395" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.56:50052\" within 1s: context deadline exceeded" Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.716586 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.724219 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.788576 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" podUID="4cf50264-0d76-46ca-b2cb-998ae3bbeb1b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.58:50052\" within 1s: context deadline exceeded" Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:51:01.788735 5490 logging.go:55] [core] [Channel #3484 SubChannel #3487]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.58:50052", ServerName: "10.133.0.58:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.58:50052: operation was canceled" Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.788793 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" podUID="bae51678-b17d-4a9c-98da-77d3ecf0d6ed" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.788845 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.788927 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" podUID="fd89b915-fe3a-4a8a-aaf5-6927987b2e0d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.22:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:51:01.798774 5490 logging.go:55] [core] [Channel #3481 SubChannel #3486]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.56:50052", ServerName: "10.133.0.56:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.56:50052: operation was canceled" Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:51:01.798798 5490 logging.go:55] [core] [Channel #3483 SubChannel #3490]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.98:50052", ServerName: "10.133.0.98:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.98:50052: operation was canceled" Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.800754 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.306224953s: [/var/lib/containers/storage/overlay/a44a874ec0085e8d4255949d87ae5c7a1486b2b9dd9c70b9902e94d80c749a81/diff /var/log/pods/e2e-models-e2e-worker-w4-dda5cb_distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5_1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.822549 5490 patch_prober.go:29] interesting pod/thanos-querier-b949d9bf6-8b6xc container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.10:9091/-/ready\": context deadline exceeded" start-of-body= Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.822594 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/ready\": context deadline exceeded" Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.887136 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" podUID="5d9ae28c-045f-456f-bbd7-8105b6e0077d" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.887223 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-586ddd746f-nv8gm" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.905030 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.235146421s: [/var/lib/containers/storage/overlay/c81dbbb6bb299b324b4560fb375259db75abcc470c453fdd9239fc9164d79480/diff /var/log/pods/openshift-ingress_e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6_b0596bd0-518a-45ab-8cc1-8ba75268ef6a/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.907044 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" podUID="6f4d14ca-ec36-4daa-8971-93c7dd709b48" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.39:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.915660 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" podUID="253719d9-0fac-4d41-bfc3-d4d359ba04fe" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.135:8443/health\": context deadline exceeded" Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.918114 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.568673739s: [/var/lib/containers/storage/overlay/553fe738bd4591523e23b5ada9f939bb6f5aee3d700c51f40149eee2f6cc2fa4/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-t9bq7_6ffcf283-0d79-422e-82be-8b34db814efd/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:01.919235 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.557516812s: [/var/lib/containers/storage/overlay/e5e3f60a602f55b84461b3e7b2af58f2dbb1e0107603a08c1a345efdabc2ed73/diff /var/log/pods/openshift-ingress_e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz_db4dcf84-a219-4ff2-b4ff-5549ef2fafe5/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:51:02.282433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.215240 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.794293721s: [/var/lib/containers/storage/overlay/55fff06e2f5611dc6c049f66754026723e546c20295d72ed62b77348c3fd8f45/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr_669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:51:02.899768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.741984 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:02.899768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.813891 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05bbcb81-55ce-4387-89c3-f6350fe8636f-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:02.899768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.814079 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:02.899768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.814123 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:02.899768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.814201 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:02.899768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.814246 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:02.899768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.814427 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:02.899768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.814462 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:02.899768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.814574 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcs5h\" (UniqueName: \"kubernetes.io/projected/05bbcb81-55ce-4387-89c3-f6350fe8636f-kube-api-access-jcs5h\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:02.899768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.814606 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.904579 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.915905 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.915952 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.916011 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcs5h\" (UniqueName: \"kubernetes.io/projected/05bbcb81-55ce-4387-89c3-f6350fe8636f-kube-api-access-jcs5h\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.916030 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.916118 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05bbcb81-55ce-4387-89c3-f6350fe8636f-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.916241 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.916279 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.916338 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:02.916376 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.056537 5490 patch_prober.go:29] interesting pod/thanos-querier-b949d9bf6-8b6xc container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.10:9091/-/ready\": context deadline exceeded" start-of-body= Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.056587 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/ready\": context deadline exceeded" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.081293 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.081547 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.081607 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.081717 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.081787 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.081873 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.083062 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.083146 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.083171 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.083241 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.083294 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.083312 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.083377 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.083425 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.083465 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.083483 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.083545 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.096883 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" podUID="efd854cd-40a6-4f9f-bd19-cd5d19512dba" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.24:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.120274 5490 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-5kbd8 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.11:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.120324 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" podUID="8bae2714-9c1b-44db-a6ef-b0b008cf1fa2" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.133.0.11:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.165443 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.165491 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" podUID="f57cc204-16b4-4735-82c1-7cee96c1704d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.174810 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" podUID="efd854cd-40a6-4f9f-bd19-cd5d19512dba" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.24:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.178814 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" podUID="ceb68ad8-3da3-4469-9207-cb473d1dad9e" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.193828 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" podUID="c9d776b1-fabe-4396-adb1-030d1edad90c" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.193838 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" podUID="9e7f6e11-0382-4897-88e3-1d4283f5bed3" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.217860 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" podUID="55f29950-78b4-4a52-818d-dc235e4631fc" containerName="proxy" probeResult="failure" output="Get \"https://10.133.0.36:8443/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.371805 5490 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ttmkb container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.371853 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" podUID="400b676a-a3a0-459d-9da6-8d30b2d3b497" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.382781 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.394410 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.418767 5490 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ttmkb container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.418814 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" podUID="400b676a-a3a0-459d-9da6-8d30b2d3b497" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.436582 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05bbcb81-55ce-4387-89c3-f6350fe8636f-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.436846 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.436841 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.436939 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:03.765740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.462802 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-586ddd746f-nv8gm" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:06.245223 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:03.905731 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 03:51:06.245223 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:04.111790 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" podUID="30874c80-e957-4c9a-ab8f-0616de567c3d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:06.245223 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:04.401815 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-6r4mx" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.32:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:06.245223 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:04.663880 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" podUID="fd89b915-fe3a-4a8a-aaf5-6927987b2e0d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.22:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:06.245223 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:04.747323 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:51:06.245223 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:05.351822 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:06.245223 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:05.675786 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:06.810024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:06.469183 5490 patch_prober.go:29] interesting pod/thanos-querier-b949d9bf6-8b6xc container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.10:9091/-/ready\": context deadline exceeded" start-of-body= Oct 31 03:51:06.810024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:06.469517 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/ready\": context deadline exceeded" Oct 31 03:51:06.810024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:06.473741 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:06.810024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:06.474381 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:06.810024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:06.474494 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:06.810024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:06.480312 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcs5h\" (UniqueName: \"kubernetes.io/projected/05bbcb81-55ce-4387-89c3-f6350fe8636f-kube-api-access-jcs5h\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:06.810024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:06.515748 5490 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ttmkb container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:06.810024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:06.515814 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" podUID="400b676a-a3a0-459d-9da6-8d30b2d3b497" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:07.226995 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" podUID="85995108-6b7b-4e56-8260-00b5347d6e6a" containerName="postgres" probeResult="failure" output="command timed out" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:07.481268 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:07.481528 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:07.482120 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:07.482246 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:07.482486 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:07.482711 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:07.483374 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:07.483435 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:07.484147 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:07.484156 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:07.497419 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" podUID="775a8940-cb22-4774-a43f-c6e7c5364354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:07.497539 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" podUID="f64d3e84-440a-4298-9230-ae9e0063c962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:07.526741 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" podUID="80180652-fb4f-4976-956d-25a47344d581" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:07.526831 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:07.527927 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:07.605474 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.404793 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl"] Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.424167 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" podUID="775a8940-cb22-4774-a43f-c6e7c5364354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.424656 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.425597 5490 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-5kbd8 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.11:9443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.425651 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" podUID="8bae2714-9c1b-44db-a6ef-b0b008cf1fa2" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.133.0.11:9443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.465840 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.481750 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.481966 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.482951 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.483055 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.484031 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.484259 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.485089 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.485145 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.486200 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.497702 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" podUID="f64d3e84-440a-4298-9230-ae9e0063c962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.497783 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" podUID="775a8940-cb22-4774-a43f-c6e7c5364354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.513142 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" podUID="55f29950-78b4-4a52-818d-dc235e4631fc" containerName="proxy" probeResult="failure" output="Get \"https://10.133.0.36:8443/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.527462 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.527611 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" podUID="80180652-fb4f-4976-956d-25a47344d581" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.528570 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" podUID="85995108-6b7b-4e56-8260-00b5347d6e6a" containerName="postgres" probeResult="failure" output="command timed out" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.528627 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="c835066f-39c8-42b4-9f99-1055495bcc69" containerName="prometheus" probeResult="failure" output="command timed out" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.528531 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="c835066f-39c8-42b4-9f99-1055495bcc69" containerName="prometheus" probeResult="failure" output="command timed out" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.606543 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.872570 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.891924 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.872689 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.893332 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.893389 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.895611 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.899372 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.900521 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.900575 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.900624 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.902873 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.902963 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.903045 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.903097 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.903158 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.904913 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.907845 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:08.907910 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:09.482783 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:09.482784 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:09.483837 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:09.483844 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:09.484955 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:09.484959 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:09.486049 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:09.486056 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:09.487142 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:09.498328 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" podUID="775a8940-cb22-4774-a43f-c6e7c5364354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:09.498336 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" podUID="f64d3e84-440a-4298-9230-ae9e0063c962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:09.527626 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:09.527724 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" podUID="80180652-fb4f-4976-956d-25a47344d581" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:09.553110 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" podUID="604d3706-e3e3-4fe2-a9bf-5472a41ee2f0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.107:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:09.607544 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:09.640008 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" podUID="288a58cd-9286-422b-bff8-15da47bc1a24" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.60:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:09.892283 5490 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-5kbd8 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.11:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:09.892361 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" podUID="8bae2714-9c1b-44db-a6ef-b0b008cf1fa2" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.133.0.11:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:09.908629 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" podUID="55f29950-78b4-4a52-818d-dc235e4631fc" containerName="proxy" probeResult="failure" output="Get \"https://10.133.0.36:8443/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:10.245573 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" podUID="76c026b4-65c7-4d57-9945-d4ebfb66ca70" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.99:8443/health\": context deadline exceeded" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:10.264845 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" podUID="253719d9-0fac-4d41-bfc3-d4d359ba04fe" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.135:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:10.483719 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:10.484747 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:10.485916 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:10.499199 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" podUID="775a8940-cb22-4774-a43f-c6e7c5364354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:10.608359 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:10.699548 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" podUID="6f4d14ca-ec36-4daa-8971-93c7dd709b48" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.39:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:11.436222 5490 patch_prober.go:29] interesting pod/thanos-querier-b949d9bf6-8b6xc container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.10:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:11.436305 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:11.586902 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" podUID="e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.101:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:11.609056 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:11.613227 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" podUID="669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.57:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:11.616353 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" podUID="669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.57:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:11.662840 5490 patch_prober.go:29] interesting pod/dns-default-vcg5q container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:13.593188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:11.662973 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-vcg5q" podUID="062e4d3d-4460-4412-93d3-22681a9a74e6" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:26.991862 5490 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-t7x7m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.160281 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" podUID="288a58cd-9286-422b-bff8-15da47bc1a24" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.60:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.230606 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" podUID="1303047d-c782-41c7-bc35-d3298c9ef224" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.62:8443/health\": context deadline exceeded" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.304754 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.362618 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.362645 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" podUID="da913fbf-0ba8-4607-be4c-5197a22d1a7d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.133:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.362704 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" podUID="dcfb42f0-a328-4179-9a5d-ac6669e132a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.424832 5490 patch_prober.go:29] interesting pod/dns-default-vcg5q container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.424894 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-vcg5q" podUID="062e4d3d-4460-4412-93d3-22681a9a74e6" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.464301 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" podUID="decb2697-571c-48e2-88b3-84a03dfbd80a" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.96:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.465588 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.788036 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" podUID="604d3706-e3e3-4fe2-a9bf-5472a41ee2f0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.107:8443/health\": context deadline exceeded" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.796842 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" podUID="30874c80-e957-4c9a-ab8f-0616de567c3d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/healthz\": dial tcp 10.133.0.28:8081: connect: connection refused" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.798288 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" podUID="531c8516-8a62-4add-a9bb-15b02ebb6f5d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.61:8443/health\": context deadline exceeded" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.802080 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": context deadline exceeded" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.803355 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" podUID="55f29950-78b4-4a52-818d-dc235e4631fc" containerName="proxy" probeResult="failure" output="Get \"https://10.133.0.36:8443/health\": context deadline exceeded" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.803366 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.803452 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.803492 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" podUID="b0b3f9b0-9f95-4556-87a2-0d7473db75f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.803574 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.817801609s: [/var/lib/containers/storage/overlay/a3ecc6bf993fa1187e40fa0ec6e71256675eae76102bfc7e2c62e6d69188c6a9/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-k9bcw_6e563544-8d00-4f0f-bd8a-c0cfc2c5c666/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.803827 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.375823814s: [/var/lib/containers/storage/overlay/3d5b2d43e79741b7bea13284dafd83c00bd24a5bc988dd02deea51f59824123e/diff /var/log/pods/odh-ai-gateway-infra_maas-api-78fdcdcc4-gp5mh_6f4d14ca-ec36-4daa-8971-93c7dd709b48/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.804374 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" podUID="604d3706-e3e3-4fe2-a9bf-5472a41ee2f0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.107:8443/health\": http2: client conn could not be established" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.808935 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" podUID="efd854cd-40a6-4f9f-bd19-cd5d19512dba" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.24:8081/healthz\": dial tcp 10.133.0.24:8081: connect: connection refused" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.809067 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" podUID="85995108-6b7b-4e56-8260-00b5347d6e6a" containerName="postgres" probeResult="failure" output="command timed out" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.829840 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" podUID="30874c80-e957-4c9a-ab8f-0616de567c3d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/readyz\": dial tcp 10.133.0.28:8081: connect: connection refused" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.830959 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.831180 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824618ba-ce21-4eeb-bf27-f1a861327863-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.831220 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" podUID="30874c80-e957-4c9a-ab8f-0616de567c3d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/readyz\": dial tcp 10.133.0.28:8081: connect: connection refused" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.831319 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.831354 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwkln\" (UniqueName: \"kubernetes.io/projected/824618ba-ce21-4eeb-bf27-f1a861327863-kube-api-access-fwkln\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.831483 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/824618ba-ce21-4eeb-bf27-f1a861327863-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.831609 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824618ba-ce21-4eeb-bf27-f1a861327863-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.831708 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.831818 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.831909 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.844042 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-586ddd746f-nv8gm" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.845423 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-586ddd746f-nv8gm" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.845484 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd"] Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:10.484766 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:10.485903 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.858557 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:10.487003 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:10.488083 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:10.528749 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.870959 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl"] Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.871021 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/odh-model-controller-769875db56-6r4mx"] Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.880245 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" podUID="efd854cd-40a6-4f9f-bd19-cd5d19512dba" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.24:8081/readyz\": dial tcp 10.133.0.24:8081: connect: connection refused" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.881244 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" podUID="efd854cd-40a6-4f9f-bd19-cd5d19512dba" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.24:8081/readyz\": dial tcp 10.133.0.24:8081: connect: connection refused" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:51:27.892427 5490 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57cc204_16b4_4735_82c1_7cee96c1704d.slice/crio-7a99bdb12b9060c5cf02942ab341f715cc430a9f268d937a3e8c0ce4bb1af3ff\": RecentStats: unable to find data in memory cache]" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.900343 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.900395 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.914715 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.407936122s: [/var/lib/containers/storage/overlay/ce5bce79fb1c98e2613e4ed83c1062dcf4c1ccd4cdbcfaa6bcaa2d7b6990dd22/diff /var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-t7x7m_decb2697-571c-48e2-88b3-84a03dfbd80a/discovery/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:27.920935 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.404810799s: [/var/lib/containers/storage/overlay/5dac8707dca68432f50b777d58729598ea5cb1d947c6f853cb91b3a5274dbae8/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg_b8353165-d9b4-445a-9a11-5fdf04307860/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.042821 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.074044316s: [/var/lib/containers/storage/overlay/6dfb813c329b17081967f32fcbc805f5785d267bd125f14b77b90dfd6506b196/diff /var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-wnnv7_ce529bc9-a44d-4808-8d2c-996e14f88a54/manager/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.043268 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.195694484s: [/var/lib/containers/storage/overlay/5aca3987237570c2137f9a52c95793fed669721aa96ec4c373b98d7d88e07d28/diff /var/log/pods/openshift-ingress_e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v_ca122c09-e327-4371-9879-0870b37db17f/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.050218 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.050384 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824618ba-ce21-4eeb-bf27-f1a861327863-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.050469 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.050510 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwkln\" (UniqueName: \"kubernetes.io/projected/824618ba-ce21-4eeb-bf27-f1a861327863-kube-api-access-fwkln\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.050607 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/824618ba-ce21-4eeb-bf27-f1a861327863-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.050752 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824618ba-ce21-4eeb-bf27-f1a861327863-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.050800 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.050850 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.050915 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.146061 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" podUID="f64d3e84-440a-4298-9230-ae9e0063c962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.281893 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.282553 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" podUID="9e7f6e11-0382-4897-88e3-1d4283f5bed3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.49:50052\" within 1s: context deadline exceeded" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:51:28.339917 5490 logging.go:55] [core] [Channel #3521 SubChannel #3524]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.49:50052", ServerName: "10.133.0.49:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:60658->10.133.0.49:50052: use of closed network connection" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.376248 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.377746 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" podUID="55f29950-78b4-4a52-818d-dc235e4631fc" containerName="proxy" probeResult="failure" output="Get \"https://10.133.0.36:8443/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.377999 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" podUID="ceb68ad8-3da3-4469-9207-cb473d1dad9e" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.55:50052\" within 1s: context deadline exceeded" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.379143 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" podUID="5d9ae28c-045f-456f-bbd7-8105b6e0077d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.54:50052\" within 1s: context deadline exceeded" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.382619 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.382663 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" podUID="dacc99d6-e9da-46e0-841d-14c5f54dd8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.383790 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.383841 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" podUID="5123bf1e-0ad6-4406-86b0-2b9a9e025a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.388102 5490 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-5kbd8 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.11:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.388148 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" podUID="8bae2714-9c1b-44db-a6ef-b0b008cf1fa2" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.133.0.11:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.389592 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.389627 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" podUID="ca122c09-e327-4371-9879-0870b37db17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.389885 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.389927 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.392036 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.394297 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" podUID="c9d776b1-fabe-4396-adb1-030d1edad90c" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.48:50052\" within 1s: context deadline exceeded" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.395806 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.421399 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" podUID="cdf49e86-d2b1-4888-8475-8a85dd69a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.400935 5490 patch_prober.go:29] interesting pod/thanos-querier-b949d9bf6-8b6xc container/kube-rbac-proxy-web namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.133.0.10:9091/-/healthy\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.421737 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/healthy\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:51:28.421324 5490 logging.go:55] [core] [Channel #3526 SubChannel #3528]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.55:50052", ServerName: "10.133.0.55:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:58994->10.133.0.55:50052: use of closed network connection" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.422041 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-586ddd746f-nv8gm" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:51:28.422834 5490 logging.go:55] [core] [Channel #3525 SubChannel #3530]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.54:50052", ServerName: "10.133.0.54:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:47480->10.133.0.54:50052: use of closed network connection" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.426758 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.432587 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.432629 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" podUID="f57cc204-16b4-4735-82c1-7cee96c1704d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:51:28.435583 5490 logging.go:55] [core] [Channel #3520 SubChannel #3523]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.48:50052", ServerName: "10.133.0.48:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:39068->10.133.0.48:50052: use of closed network connection" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.439548 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.439590 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" podUID="101e77ad-e8f5-4222-83ee-982ac3954dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.440076 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.440144 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" podUID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.441878 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" podUID="fd89b915-fe3a-4a8a-aaf5-6927987b2e0d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.22:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.785958 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.786036 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.789521 5490 patch_prober.go:29] interesting pod/thanos-querier-b949d9bf6-8b6xc container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.10:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.789574 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.826368 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-t9bq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.826418 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" podUID="6ffcf283-0d79-422e-82be-8b34db814efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.832797 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.841356 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.859485 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" podUID="775a8940-cb22-4774-a43f-c6e7c5364354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.868826 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.872813 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" podUID="42e53881-926d-482d-9557-a86d3ae3f9d9" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.890809 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" podUID="fd89b915-fe3a-4a8a-aaf5-6927987b2e0d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.22:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:28.976124 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" podUID="30874c80-e957-4c9a-ab8f-0616de567c3d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/readyz\": dial tcp 10.133.0.28:8081: connect: connection refused" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:29.148312 5490 trace.go:236] Trace[956514984]: "Calculate volume metrics of data for pod odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" (31-Oct-2026 03:51:27.839) (total time: 1308ms): Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: Trace[956514984]: [1.308336487s] [1.308336487s] END Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:29.336809 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" podUID="efd854cd-40a6-4f9f-bd19-cd5d19512dba" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.24:8081/readyz\": dial tcp 10.133.0.24:8081: connect: connection refused" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:29.424344 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:29.841571 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:29.874805 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:29.892539 5490 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-5kbd8 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.11:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:29.892592 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" podUID="8bae2714-9c1b-44db-a6ef-b0b008cf1fa2" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.133.0.11:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:29.906768 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" podUID="55f29950-78b4-4a52-818d-dc235e4631fc" containerName="proxy" probeResult="failure" output="Get \"https://10.133.0.36:8443/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:29.911858 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:29.931841 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" podUID="fd89b915-fe3a-4a8a-aaf5-6927987b2e0d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.22:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:30.337280 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-586ddd746f-nv8gm" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:30.438782 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:30.439008 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824618ba-ce21-4eeb-bf27-f1a861327863-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:30.688800 5490 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ttmkb container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:30.688867 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" podUID="400b676a-a3a0-459d-9da6-8d30b2d3b497" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:30.689270 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:30.841979 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:30.896822 5490 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ttmkb container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:30.896920 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" podUID="400b676a-a3a0-459d-9da6-8d30b2d3b497" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:32.663020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:30.954869 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:30.972823 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" podUID="fd89b915-fe3a-4a8a-aaf5-6927987b2e0d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.22:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:30.979893 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:30.979893 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:31.436092 5490 patch_prober.go:29] interesting pod/thanos-querier-b949d9bf6-8b6xc container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.10:9091/-/ready\": context deadline exceeded" start-of-body= Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:31.436159 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/ready\": context deadline exceeded" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:31.843050 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:32.506796 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:32.843237 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:32.870822 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.038977 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.040949 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.101952 5490 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ttmkb container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.102011 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" podUID="400b676a-a3a0-459d-9da6-8d30b2d3b497" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.388796 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824618ba-ce21-4eeb-bf27-f1a861327863-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:51:34.411216 5490 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="6.54s" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.411251 5490 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.424776 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.424928 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.425392 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.425451 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.425491 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.444582 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.444649 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.444910 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.444978 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.445044 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.445082 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.445124 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.445164 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.445239 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.445330 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.445400 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.445448 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.445486 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.613905 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/824618ba-ce21-4eeb-bf27-f1a861327863-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.738500 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:34.739259 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" podUID="30874c80-e957-4c9a-ab8f-0616de567c3d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/healthz\": dial tcp 10.133.0.28:8081: connect: connection refused" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:35.079938 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-6r4mx" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.32:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:35.160727 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:35.286215 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:35.295541 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwkln\" (UniqueName: \"kubernetes.io/projected/824618ba-ce21-4eeb-bf27-f1a861327863-kube-api-access-fwkln\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:35.295867 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:35.670032 5490 patch_prober.go:29] interesting pod/dns-default-vcg5q container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:35.670096 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-vcg5q" podUID="062e4d3d-4460-4412-93d3-22681a9a74e6" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:35.941208 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" podUID="efd854cd-40a6-4f9f-bd19-cd5d19512dba" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.24:8081/readyz\": dial tcp 10.133.0.24:8081: connect: connection refused" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:35.943072 5490 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="cert-manager-webhook" containerStatusID={"Type":"cri-o","ID":"ec00228885dd52018a1ab0bc0d3946855d0149c08a6916c4ed6781db4fbed4ba"} pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" containerMessage="Container cert-manager-webhook failed liveness probe, will be restarted" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:35.943139 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" containerID="cri-o://ec00228885dd52018a1ab0bc0d3946855d0149c08a6916c4ed6781db4fbed4ba" gracePeriod=30 Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:36.686757 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" podUID="efd854cd-40a6-4f9f-bd19-cd5d19512dba" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.24:8081/healthz\": dial tcp 10.133.0.24:8081: connect: connection refused" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:36.686844 5490 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:36.953836 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" podUID="55f29950-78b4-4a52-818d-dc235e4631fc" containerName="proxy" probeResult="failure" output="Get \"https://10.133.0.36:8443/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:36.982812 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" podUID="fd89b915-fe3a-4a8a-aaf5-6927987b2e0d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.22:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:37.096770 5490 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ttmkb container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:37.096823 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" podUID="400b676a-a3a0-459d-9da6-8d30b2d3b497" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:38.178249 5490 patch_prober.go:29] interesting pod/dns-default-vcg5q container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.133.0.6:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:38.178291 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-vcg5q" podUID="062e4d3d-4460-4412-93d3-22681a9a74e6" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.6:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:39.296520 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:39.369792 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:39.513666 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:39.521194 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:39.521251 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:39.756557 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" podUID="f64d3e84-440a-4298-9230-ae9e0063c962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 03:51:40.167265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:39.815646 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" podUID="80180652-fb4f-4976-956d-25a47344d581" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:51:44.073237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:51:42.835745 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" podUID="669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.57:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:07.021746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:06.829731 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.008352891s: [/var/lib/containers/storage/overlay/57017cf4a74a77b9f4fd0ecf8711d9f6c69dcfb1452f3c5f95d63ca539a4634e/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5_531c8516-8a62-4add-a9bb-15b02ebb6f5d/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:07.021746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:06.829893 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.106894032s: [/var/lib/containers/storage/overlay/a44a874ec0085e8d4255949d87ae5c7a1486b2b9dd9c70b9902e94d80c749a81/diff /var/log/pods/e2e-models-e2e-worker-w4-dda5cb_distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5_1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 31 03:52:07.021746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:06.989229 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" podUID="b7c0d839-b3a4-463b-a3a8-3733609d4b89" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.134:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:07.021746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:06.989448 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" podUID="604d3706-e3e3-4fe2-a9bf-5472a41ee2f0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.107:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:07.021746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:06.989523 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:07.021746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:06.989560 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" podUID="531c8516-8a62-4add-a9bb-15b02ebb6f5d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.61:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:07.021746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:06.991703 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" podUID="e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.101:8443/health\": context deadline exceeded" Oct 31 03:52:07.021746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:07.008627 5490 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ttmkb container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:07.021746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:07.008708 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" podUID="400b676a-a3a0-459d-9da6-8d30b2d3b497" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:07.021746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:07.011795 5490 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-5kbd8 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.11:9443/health\": context deadline exceeded" start-of-body= Oct 31 03:52:07.021746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:07.011850 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" podUID="8bae2714-9c1b-44db-a6ef-b0b008cf1fa2" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.133.0.11:9443/health\": context deadline exceeded" Oct 31 03:52:07.021746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:07.014565 5490 patch_prober.go:29] interesting pod/network-check-target-4tzpt container/network-check-target-container namespace/openshift-network-diagnostics: Readiness probe status=failure output="Get \"http://10.133.0.5:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:07.021746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:07.014605 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-network-diagnostics/network-check-target-4tzpt" podUID="d92bdf4f-f9e4-424c-9c87-ec015574ff1a" containerName="network-check-target-container" probeResult="failure" output="Get \"http://10.133.0.5:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:07.021746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:07.023144 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" podUID="6f4d14ca-ec36-4daa-8971-93c7dd709b48" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.39:8443/health\": context deadline exceeded" Oct 31 03:52:07.021746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:07.127617 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" podUID="fd89b915-fe3a-4a8a-aaf5-6927987b2e0d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.22:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:07.021746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:07.166760 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.074632885s: [/var/lib/containers/storage/overlay/7f6d3194413178481ce2ccceac907da1e3c81744526e3d50085b9a47d96578e8/diff /var/log/pods/openshift-ingress_e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996_1ce35023-5f08-4356-8e09-4328952c6040/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:07.021746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:07.311473 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" podUID="30874c80-e957-4c9a-ab8f-0616de567c3d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/readyz\": dial tcp 10.133.0.28:8081: connect: connection refused" Oct 31 03:52:07.021746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:07.313379 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-insights/insights-runtime-extractor-zngp8" podUID="08da34c7-c69a-4eb9-848a-399cab14840e" containerName="extractor" probeResult="failure" output="command timed out" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:07.359478 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" podUID="253719d9-0fac-4d41-bfc3-d4d359ba04fe" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.135:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:07.359551 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" podUID="da913fbf-0ba8-4607-be4c-5197a22d1a7d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.133:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:07.360193 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" podUID="775a8940-cb22-4774-a43f-c6e7c5364354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:07.688089 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" podUID="efd854cd-40a6-4f9f-bd19-cd5d19512dba" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.24:8081/readyz\": dial tcp 10.133.0.24:8081: connect: connection refused" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:07.860306 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" podUID="30874c80-e957-4c9a-ab8f-0616de567c3d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/healthz\": dial tcp 10.133.0.28:8081: connect: connection refused" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:07.860870 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" podUID="76c026b4-65c7-4d57-9945-d4ebfb66ca70" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.99:8443/health\": context deadline exceeded" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:07.876346 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" podUID="e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.101:8443/health\": EOF" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:07.892046 5490 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-t7x7m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:07.892084 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" podUID="decb2697-571c-48e2-88b3-84a03dfbd80a" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.96:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.159175 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.159995 5490 patch_prober.go:29] interesting pod/thanos-querier-b949d9bf6-8b6xc container/kube-rbac-proxy-web namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.133.0.10:9091/-/healthy\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.160045 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/healthy\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.160362 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.160390 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" podUID="a8a2a69d-e4b7-4338-a7b4-22042b94432f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.774519 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.774637 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" podUID="fd89b915-fe3a-4a8a-aaf5-6927987b2e0d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.22:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.774800 5490 patch_prober.go:29] interesting pod/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.774847 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" podUID="015afcb4-d156-4163-a342-f84471c9dd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.780922 5490 trace.go:236] Trace[1613948825]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" (31-Oct-2026 03:52:07.016) (total time: 1097ms): Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: Trace[1613948825]: [1.097603595s] [1.097603595s] END Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.781188 5490 trace.go:236] Trace[1196001309]: "Calculate volume metrics of openshift-service-ca for pod odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" (31-Oct-2026 03:52:07.056) (total time: 1082ms): Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: Trace[1196001309]: [1.082158222s] [1.082158222s] END Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.781360 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" podUID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.783903 5490 trace.go:236] Trace[1299822402]: "Calculate volume metrics of openshift-service-ca for pod odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" (31-Oct-2026 03:52:06.857) (total time: 1926ms): Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: Trace[1299822402]: [1.92609347s] [1.92609347s] END Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.722220 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" podUID="55f29950-78b4-4a52-818d-dc235e4631fc" containerName="proxy" probeResult="failure" output="Get \"https://10.133.0.36:8443/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.784090 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.810310 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.810396 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.825941 5490 trace.go:236] Trace[36893317]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" (31-Oct-2026 03:51:39.742) (total time: 28347ms): Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: Trace[36893317]: [28.347457073s] [28.347457073s] END Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.827213 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.827286 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.827646 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": context deadline exceeded" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.827780 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.828117 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" podUID="80180652-fb4f-4976-956d-25a47344d581" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.828351 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.867889 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-586ddd746f-nv8gm" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.867842 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.867983 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" podUID="1ce35023-5f08-4356-8e09-4328952c6040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.868813 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.868862 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" podUID="dcfb42f0-a328-4179-9a5d-ac6669e132a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.868923 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.868938 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" podUID="cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.868989 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.869010 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:08.884336 5490 trace.go:236] Trace[535854209]: "Calculate volume metrics of tls-certs for pod ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" (31-Oct-2026 03:52:04.966) (total time: 2358ms): Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: Trace[535854209]: [2.358597722s] [2.358597722s] END Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:09.658803 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" podUID="531c8516-8a62-4add-a9bb-15b02ebb6f5d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.61:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:09.802355 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" podUID="da913fbf-0ba8-4607-be4c-5197a22d1a7d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.133:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:09.831830 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" podUID="b7c0d839-b3a4-463b-a3a8-3733609d4b89" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.134:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:10.012827 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" podUID="d2a15588-0492-4ccc-96cc-ab09e1e1402d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.59:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:11.683808 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" podUID="1303047d-c782-41c7-bc35-d3298c9ef224" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.62:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:11.821328 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" podUID="c809aea3-9195-4f90-9ffb-ca153af1a5be" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.108:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:11.831476 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" podUID="288a58cd-9286-422b-bff8-15da47bc1a24" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.60:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:12.037770 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" podUID="6f4d14ca-ec36-4daa-8971-93c7dd709b48" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.39:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:34.793457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:12.147412 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" podUID="669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.57:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.573740 5490 patch_prober.go:29] interesting pod/thanos-querier-b949d9bf6-8b6xc container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.10:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.648305 5490 ???:1] "http: TLS handshake error from 10.0.129.132:48714: EOF" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.649039 5490 ???:1] "http: TLS handshake error from 10.0.141.153:51935: EOF" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:52:45.661140 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="86d75c08ecd5493020526daadaf8a8ee9ee5987fb034c16e3a91392ef7ed9ada" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.790210 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" podUID="d2a15588-0492-4ccc-96cc-ab09e1e1402d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.59:8443/health\": net/http: TLS handshake timeout" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.849229 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.872176 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.744592171s: [/var/lib/containers/storage/overlay/b3115c24c43be22f82075c15c57406970f9f322890b1216fcc85c8724d180071/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.872221 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.744647874s: [/var/lib/containers/storage/overlay/133d74429b5ac071102bdf3f5ca952a39a3a06c070bd0ee00fb731606fab984e/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.872245 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.744661161s: [/var/lib/containers/storage/overlay/3edbd53ff7c3e2db6d434a596d7197dd44bd7ba41a1a121712f6f7f9a0a530f2/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.872279 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.8588317s: [/var/lib/containers/storage/overlay/01cba254af9216ae987296facb36a38be1d32eb3332de50bbaef339c9ea91da6/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.872389 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.744801264s: [/var/lib/containers/storage/overlay/10ec65371f848b12073db4805a9b7b53ecc4646eb6a8f92e4957e96ae53f4e16/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.873495 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.746858702s: [/var/lib/containers/storage/overlay/a87a6b7d23c9256b013907e23a11ec6040305e0327057832b1c1fc2005b0dc06/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.873922 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-6r4mx" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.32:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.873988 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.874005 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" podUID="cdf49e86-d2b1-4888-8475-8a85dd69a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.874042 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.874060 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" podUID="b0b3f9b0-9f95-4556-87a2-0d7473db75f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.874114 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.74657122s: [/var/lib/containers/storage/overlay/a89cffaca2e94209c4a1eb800a3122c13cf8acf3cc9bf713e118f51e011b3129/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.874553 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/maas-controller-586ddd746f-nv8gm" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.874658 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.74711075s: [/var/lib/containers/storage/overlay/40545f2172e148f38261d23823ee8e94402ddb49fe702ac1cf19470eab52fc90/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.874719 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.732932284s: [/var/lib/containers/storage/overlay/80c0c716146da823694d60651ffe8842f4f368241ce10b4ad4e7d85868eb9064/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.874890 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.733101485s: [/var/lib/containers/storage/overlay/14fce3c8b9f1b6ea430917a651a1b96a9c8267e9f715201effdd4239be0076b6/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.874927 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.733137211s: [/var/lib/containers/storage/overlay/e7815039fab1e054df794ed43face9365f8cee418a3d9d142c00e8d2a8440b77/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875064 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.733271583s: [/var/lib/containers/storage/overlay/b57ed64a4dff40f6b15820fc5b07db861d686ede6e250081823b4e52b2125f6e/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875268 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.733471558s: [/var/lib/containers/storage/overlay/2a552eed271789ef7cc0ffc04af7bea1f89527348fc2bd1ecb8c2f2fd5fc7774/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875307 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.733509069s: [/var/lib/containers/storage/overlay/0a623c6822a00610cd6dc5f2b2b1cedee68accb34e16ccfe996cc747499f8969/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875343 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.733544566s: [/var/lib/containers/storage/overlay/bd934b09c6c317d50255e8fa7a6dbb42ed8721260b5e9eafc7560931319cf7a2/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875551 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.733744414s: [/var/lib/containers/storage/overlay/da4f056976a9bccbc0f12bc003390e91475ac88d255466a9a9a6b9d501ff4645/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875587 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.733780298s: [/var/lib/containers/storage/overlay/0e24d8fd1d2aa5f04638488c56d1f2be91c079676584bb0163c3af4864d321ed/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875613 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.733805251s: [/var/lib/containers/storage/overlay/b118968fba82d3613ed21d5eb8d2213337c5377f2056fe3b41dc86d15155646c/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875639 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.733830512s: [/var/lib/containers/storage/overlay/ad680ebaa62988f4d094a44a91c9f79cbfd6f38bced2c4077eba0b03b9b83d00/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875662 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.733852525s: [/var/lib/containers/storage/overlay/30c5aac273d5883c21db629b4d98df0a5b06f9e60c67e4ba47bd96d8cf4e20e3/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875700 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.733888621s: [/var/lib/containers/storage/overlay/49e7d30229113951488827deebfafa892db861ac9dc7eca98626d16e1fbdcc11/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875726 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.733914081s: [/var/lib/containers/storage/overlay/0eb5ffcca854c9724d38b7f1f9bea0e04cb8f4f546668d74f9f2763d94051963/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875751 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.733937509s: [/var/lib/containers/storage/overlay/1628a9ad42d6a909300cc2b920d96439b0e343ab2a7805a60fd331600f2b353a/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875777 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.733962226s: [/var/lib/containers/storage/overlay/01d8e5d18484f8cf0ba1a849af7c7f5794b94f3b63391402a965040cd495c0b8/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875806 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.733990291s: [/var/lib/containers/storage/overlay/b7b278f9ab1ddf31d5056c5a425fceaba39a3d95289af7efdc03aa45b1e016fd/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875832 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.734014909s: [/var/lib/containers/storage/overlay/6b0d111830490333f01540eba542815190fe3812b791e293551c25603c9e6e4d/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875856 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.734032389s: [/var/lib/containers/storage/overlay/01268d515835b965577ab971300ec51cbf745ff93ef4521a5b2cf4995abbab51/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875886 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.734060818s: [/var/lib/containers/storage/overlay/41d85652ed85264c3295cf59ddef82c9a31bd00d340fc45bf69388722338c62d/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875912 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.734085606s: [/var/lib/containers/storage/overlay/18812db555995c0881aca03258a74f341a500f066d37eb4d2cf441d274376f79/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875937 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.734108969s: [/var/lib/containers/storage/overlay/a6d3e9aebc3c1a73ea6ddf76e33a312eb36b37b709ca3458919e3cce369b71d7/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875967 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.734137623s: [/var/lib/containers/storage/overlay/d60d48b8ff7dd5cf3885ff2b24958afbf419c28bbed63493ed57cb2f99b6f64f/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.875992 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.734162286s: [/var/lib/containers/storage/overlay/441b6eb1b50cb7ca3d49376d427d2052b35163f2fe9c2b4892c614c71f8eb4cf/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.876155 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.734321997s: [/var/lib/containers/storage/overlay/5e61c1d43b7ee586666ac2ccd128e0adfc3092846853216b87d140fa05288f8f/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.876183 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.734348535s: [/var/lib/containers/storage/overlay/ac9f761b248ad37ffe9a5491e2ded7e8423104294b6d510456fe6e22e99ff9f4/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.876214 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.734379049s: [/var/lib/containers/storage/overlay/3843bbfe721e2d3ee19959305fd1bd495cf0a5db4495ea61c06c528691352067/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.876820 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" podUID="c809aea3-9195-4f90-9ffb-ca153af1a5be" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.108:8443/health\": context deadline exceeded" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.876865 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" podUID="b8353165-d9b4-445a-9a11-5fdf04307860" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.105:8443/health\": context deadline exceeded" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.876887 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" podUID="1303047d-c782-41c7-bc35-d3298c9ef224" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.62:8443/health\": context deadline exceeded" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.877859 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" podUID="288a58cd-9286-422b-bff8-15da47bc1a24" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.60:8443/health\": context deadline exceeded" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.935233 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.78228791s: [/var/lib/containers/storage/overlay/4988c394c3a1e1b2dc0e4f5571219f75d881bca653f22a4d6ff24bfa310ed791/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.946250 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.144373015s: [/var/lib/containers/storage/overlay/12c70a9383412b5cd185172ceeedc66a394bc57afcb5a336c4f3e0e3284e7417/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_c835066f-39c8-42b4-9f99-1055495bcc69/prometheus/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981042 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981095 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" podUID="ca122c09-e327-4371-9879-0870b37db17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981097 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981126 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981139 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" podUID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981146 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.622247313s: [/var/lib/containers/storage/overlay/8228edc6c6bb838c65eb67f702d127147bedd7f40da8f2a1912b375f900ab1d8/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981157 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981177 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.179734956s: [/var/lib/containers/storage/overlay/7555f64ee5f95027080347060b0982a97e6d8a62989cff8f37a90cc2bc03a15e/diff /var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-55975df5cc-vtdtl_fd89b915-fe3a-4a8a-aaf5-6927987b2e0d/manager/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981208 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.6221818s: [/var/lib/containers/storage/overlay/c169bbc5bd05ab2c90def270cfc5109a0e3276e24da16b02a316850341fd7a4d/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981245 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.622217785s: [/var/lib/containers/storage/overlay/8f65339d73e57f1f258b2b0c1b59925cb636ac9fd61dbbf1810d954a961ccfbd/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981275 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.622248701s: [/var/lib/containers/storage/overlay/03e1beaea0c0a3df12d65dbb94b8cd83c6d0502030680c6db0d2d174f0be58ac/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981278 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981301 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.622273003s: [/var/lib/containers/storage/overlay/0fd42f26cd3d4a0f0d5f95626561200bb352dc37fc1b072a6f924a7371428c0b/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981299 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" podUID="101e77ad-e8f5-4222-83ee-982ac3954dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981324 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.622296079s: [/var/lib/containers/storage/overlay/7d42bd9b5b692bd5e67b02080decaf18c947a6cf20e4288a02ed4733cfa5def1/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981465 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981491 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" podUID="f57cc204-16b4-4735-82c1-7cee96c1704d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981892 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.622858307s: [/var/lib/containers/storage/overlay/4c8f79f1477063dc18286fc20a4f342c4266e0bda67634179e84d4d41d6f790b/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981921 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.622887205s: [/var/lib/containers/storage/overlay/140791275e4bdd532606a4f7a9fb8de96bd7f9f990e6b95d49216e5cddf7dd5a/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981940 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.133171772s: [/var/lib/containers/storage/overlay/0bc1ff134555d413f9bdaf5b98937c5c19ad375dc8315a833c42ea1e4c521e7e/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf_4cf50264-0d76-46ca-b2cb-998ae3bbeb1b/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981954 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.133003537s: [/var/lib/containers/storage/overlay/a2e921af46db872fc41ffb5e8a03da0c66eff158eea5a940c9769d3c8a338722/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf_c809aea3-9195-4f90-9ffb-ca153af1a5be/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981970 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.140449349s: [/var/lib/containers/storage/overlay/16082b499d5de4da2c839cda43af47b99ababdb4de41551ab1eaa6fcf2042872/diff /var/log/pods/e2e-models-e2e-worker-w4-dda5cb_premium-w4-dda5cb-kserve-796ff5cf99-g2qll_51e4007d-c749-49c9-b6b2-301e57a795b3/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981983 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.15691677s: [/var/lib/containers/storage/overlay/ecca32edc1bf21c74dae4f63070352d05b3f1dbba6b7ae8f74f576f2a8e8ba3e/diff /var/log/pods/e2e-models-e2e-worker-w3-090ebd_premium-w3-090ebd-kserve-7dc669c77-s9qp5_1c3288c4-4250-4046-921b-4382c4d3ca93/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.981998 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.126596936s: [/var/lib/containers/storage/overlay/6172f5dc7c07674b778bc24eefb6df071f98dd4936f311edb9d3e47d28c02fb3/diff /var/log/pods/openshift-ingress_e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p_5123bf1e-0ad6-4406-86b0-2b9a9e025a45/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982017 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.180167389s: [/var/lib/containers/storage/overlay/1d5cb42af67bdb5571c2eaa7f536720fbdb8179b48d6fd9d6823e6dc2a984bbb/diff /var/log/pods/openshift-ingress_e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg_209cb838-5556-47a9-96ae-e2378ad8c8b5/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982032 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.125830706s: [/var/lib/containers/storage/overlay/f4d4850fb84d31068fbcb3e5d02fce7886dfdb4fc2db887c75ccb37da749d7ce/diff /var/log/pods/openshift-ingress_e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn_015afcb4-d156-4163-a342-f84471c9dd51/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982046 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 41.005599874s: [/var/lib/containers/storage/overlay/52b5bd623f113c952a39b661abdb101eadf94a10aaccc36b496fe5240692e1bf/diff /var/log/pods/openshift-ingress_e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j_dbec9363-80a5-497b-8dd8-45b9fba41328/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982105 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.992761316s: [/var/lib/containers/storage/overlay/6193043e06af63da6b771093948fe80d960af7e091d5585d6387c3b40405f674/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n_1303047d-c782-41c7-bc35-d3298c9ef224/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982130 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.316580233s: [/var/lib/containers/storage/overlay/9cce1f7304484864923e6b2960167f44e055f351a8fc265c2e570c0152cbee65/diff /var/log/pods/openshift-image-registry_node-ca-g2mdn_55f1a18a-59ef-4146-a0e8-3fc632ab20f5/node-ca/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982147 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.134879711s: [/var/lib/containers/storage/overlay/748ada79ccd208a575c1ac6253895fcc03b0c6e7a150f588fb7cd633c50e6cb5/diff /var/log/pods/cert-manager-operator_cert-manager-operator-controller-manager-7d87bcd5fd-ggqtl_b50aa305-9fe9-48de-a7d1-e8654c325be4/cert-manager-operator/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982177 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.104610821s: [/var/lib/containers/storage/overlay/652f3bb917f7e3675abe48ebaacfa2cfdbe16a4a493cb9402eddef8230c6cafd/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982307 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.090391246s: [/var/lib/containers/storage/overlay/a36c41ca5fe3178cd6ce94cc54b0080d14ebe1e740f61d47cd1f36864a2f2952/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982332 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.09041453s: [/var/lib/containers/storage/overlay/09bdd51f64a529fb9f920db44856db7c5df805df7c3ea256f2d1409ff71cb280/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982437 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.090515143s: [/var/lib/containers/storage/overlay/e922fd5b8234e459dd9fbd74c0ab431b2e8c78f539e9247cad7529d78f4f34ff/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982467 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.090544103s: [/var/lib/containers/storage/overlay/25f654031653ebd76a9484731de6557b320179df8019b5d717cccc1eaf4ac71e/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982601 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.090674941s: [/var/lib/containers/storage/overlay/656d7ab982e2b2b911959d91aea4ee798685ecc07c3018ceaac1a06d5069abe5/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982630 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.85507817s: [/var/lib/containers/storage/overlay/03b6726514a949b7712b1d874b2847f6dd63d2ea900776b59309768d42556935/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982635 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.090705638s: [/var/lib/containers/storage/overlay/0dcd1b5f61e8d7029ca2e1bc5efec1eb1f1447f3245da7b595981ab0d11692bf/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982817 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-t9bq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982850 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" podUID="6ffcf283-0d79-422e-82be-8b34db814efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982879 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.090940007s: [/var/lib/containers/storage/overlay/0d7ec9b508b814ba1bde633899789d3f2f2448e08c3dc765229f70d109eaed64/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982901 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982910 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.090930742s: [/var/lib/containers/storage/overlay/94494744aef02e6934426a1a56af48c766ed9d3594c031ec23d6ff81a54d813e/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982920 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" podUID="db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982925 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.090946712s: [/var/lib/containers/storage/overlay/baf9c542b9896d9c762124e3c445fb2d9902d55bc1519742f6f602b0cb1760b9/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982950 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.090963334s: [/var/lib/containers/storage/overlay/36b074204d6ccbf7f3c1f2008d06919f2dce5e53576fc6830ecadfb87c8c7369/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.982976 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.090988614s: [/var/lib/containers/storage/overlay/4046f43ddbd564275e31103e9f07b1b1a8897d11f2a6c18f34cf7bd0ae9ecafa/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983003 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.091014081s: [/var/lib/containers/storage/overlay/a20e6940f0ea9a7db520b42abb82d9916dc1451e464afe581584413744f948be/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983008 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.091018156s: [/var/lib/containers/storage/overlay/38ce4c6777f145ab7399dd6002fd7448cccab9c3027f05817cf6349586c43d50/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983036 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.091042997s: [/var/lib/containers/storage/overlay/e5fc544a7fa58b3a8d8598a47584c72ec654ee6e86eb8e54e0ca7b9caa5ed6ae/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983065 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.091071441s: [/var/lib/containers/storage/overlay/183bc5aac2b9ba7bb36202e7efad21688d4168e787efe02d776f7f9ac8cbd0b8/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983091 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.091095136s: [/var/lib/containers/storage/overlay/725c426ba3cec65a3f32fbd9d4ddd55546c8307c69d3fe2716a655c46b83e109/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983119 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.091122134s: [/var/lib/containers/storage/overlay/ec798005b03958c1fc5a857a923b0bc998f8165caaf1b561ef86057d982b03e9/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983152 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.091153291s: [/var/lib/containers/storage/overlay/8db73851f4e151e935b6bbeda34384910b56f46e6770762dac8267d448981fbc/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983177 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.091176932s: [/var/lib/containers/storage/overlay/4b9e99d8c9c87cd0344363d0b584c23ba8bee896dd95c813b85a052691e41b57/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983179 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.091170546s: [/var/lib/containers/storage/overlay/2d76ae662d050dce7f5728bccb51af6d9d6c72ef228cb2f05996a521109558f1/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983205 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.091194026s: [/var/lib/containers/storage/overlay/cf85fa76dd154034dff99164d1e7c3b372d30c63c2f1cd666d99cceff41cf900/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983204 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.091193482s: [/var/lib/containers/storage/overlay/89be4864200a695869747020475a4e7db954c6c5b9453a1688efc585e180b1da/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983327 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.091312358s: [/var/lib/containers/storage/overlay/e2390e4056b9ca72b27f6e241bf5d10ff4e2521d38f0368a340b176358ac6047/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983430 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 37.873165155s: [/var/lib/containers/storage/overlay/04035c5d11597ac6ee166954fea1ac8f2db0cb33163e7488faf20fb731303759/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983487 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 37.84500517s: [/var/lib/containers/storage/overlay/830608bb6f7bb469412c465ac08f7dabe97c2d795e4933380c6d4f1e0fff5309/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983592 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 42.910199504s: [/var/lib/containers/storage/overlay/520654143d43f8c05469e9cf12f580ed3c3fdb3237a6a4039f4fe85621c02f02/diff /var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-gzjbq_c9d776b1-fabe-4396-adb1-030d1edad90c/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983608 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 42.910187798s: [/var/lib/containers/storage/overlay/9f73cbef85eb461db197f6ea81be3d8cdbabbc7066cc181b259c9ff6ecd75772/diff /var/log/pods/cert-manager_cert-manager-cainjector-5c57bbd4fc-h5bqj_6504ab4d-1fc3-4116-838a-e3708400931b/cert-manager-cainjector/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983622 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.154732651s: [/var/lib/containers/storage/overlay/95662cc85fdec0b3d5e058b73867d857ca9b05b858ee8c8c8a6e76e10d1d9964/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd_bae51678-b17d-4a9c-98da-77d3ecf0d6ed/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983638 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.179717772s: [/var/lib/containers/storage/overlay/b4b0fb17ba283afdb28f71c3c3bb9abd95afdb9f47c2d8701e88b9f91a6ba37d/diff /var/log/pods/opendatahub_maas-controller-586ddd746f-nv8gm_2238dc08-3a3e-426e-a8f7-22bec5f016c8/manager/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983654 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.149056137s: [/var/lib/containers/storage/overlay/2ec982c4cb4d0cdcd939e4cdb5ad7c61b97c311f48a994fed01ab6adb1ec7271/diff /var/log/pods/e2e-models-e2e-worker-w0-d86a77_distinct-w0-d86a77-kserve-85fb976dfd-xfxl6_d61881b1-0191-43d7-9ac2-c060af4e8f6c/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983820 5490 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ttmkb container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983847 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" podUID="400b676a-a3a0-459d-9da6-8d30b2d3b497" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983902 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.856353198s: [/var/lib/containers/storage/overlay/3cbe0df1663cbb88b0302bb71ed633bcd9f2a40981f8a983d818da5f3a0326c1/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.983922 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m6.469758731s: [/var/lib/containers/storage/overlay/66a8a64affd9e7f7a71bdb84143b965d1353f2650f0f2750422f4dd54abadbb8/diff /var/log/pods/openshift-ingress_e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h_8317b14b-20f5-4404-8f4c-bf315ec53736/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:52:45.992661 5490 webhook.go:269] Failed to make webhook authorizer request: webhook call failed: context canceled Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:52:45.992754 5490 webhook.go:269] Failed to make webhook authorizer request: webhook call failed: context canceled Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:52:45.992768 5490 server.go:358] "Authorization error" err="webhook call failed: context canceled" user="system:serviceaccount:openshift-monitoring:metrics-server" verb="get" resource="nodes" subresource="metrics" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:52:45.992720 5490 server.go:358] "Authorization error" err="webhook call failed: context canceled" user="system:serviceaccount:openshift-monitoring:metrics-server" verb="get" resource="nodes" subresource="metrics" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:52:45.992662 5490 webhook.go:269] Failed to make webhook authorizer request: webhook call failed: context canceled Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:52:45.992812 5490 server.go:358] "Authorization error" err="webhook call failed: context canceled" user="system:serviceaccount:openshift-monitoring:metrics-server" verb="get" resource="nodes" subresource="metrics" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.995513 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" podUID="9e7f6e11-0382-4897-88e3-1d4283f5bed3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.49:50052\" within 1s: context deadline exceeded" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.995936 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" podUID="bae51678-b17d-4a9c-98da-77d3ecf0d6ed" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.103:50052\" within 1s: context deadline exceeded" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:52:45.996107 5490 logging.go:55] [core] [Channel #3578 SubChannel #3582]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.49:50052", ServerName: "10.133.0.49:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.49:50052: operation was canceled" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.996922 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" podUID="e9f75494-3ba5-460d-a96d-9ab2f4ffa395" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.56:50052\" within 1s: context deadline exceeded" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:52:45.997060 5490 logging.go:55] [core] [Channel #3576 SubChannel #3583]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.103:50052", ServerName: "10.133.0.103:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.103:50052: operation was canceled" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.997832 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" podUID="c9d776b1-fabe-4396-adb1-030d1edad90c" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.48:50052\" within 1s: context deadline exceeded" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:52:45.997970 5490 logging.go:55] [core] [Channel #3579 SubChannel #3585]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.56:50052", ServerName: "10.133.0.56:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.56:50052: operation was canceled" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:45.999774 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/healthz\": dial tcp 10.133.0.26:6080: connect: connection refused" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.000776 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" podUID="30874c80-e957-4c9a-ab8f-0616de567c3d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/readyz\": dial tcp 10.133.0.28:8081: connect: connection refused" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:52:46.000990 5490 logging.go:55] [core] [Channel #3577 SubChannel #3588]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.48:50052", ServerName: "10.133.0.48:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.48:50052: operation was canceled" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.002086 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" podUID="30874c80-e957-4c9a-ab8f-0616de567c3d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/healthz\": dial tcp 10.133.0.28:8081: connect: connection refused" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.002257 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.002651 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" podUID="85995108-6b7b-4e56-8260-00b5347d6e6a" containerName="postgres" probeResult="failure" output="command timed out" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.004646 5490 patch_prober.go:29] interesting pod/dns-default-vcg5q container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.004693 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-vcg5q" podUID="062e4d3d-4460-4412-93d3-22681a9a74e6" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.007778 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="c835066f-39c8-42b4-9f99-1055495bcc69" containerName="prometheus" probeResult="failure" output="command timed out" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.007909 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="c835066f-39c8-42b4-9f99-1055495bcc69" containerName="prometheus" probeResult="failure" output="command timed out" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.007979 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" podUID="5d9ae28c-045f-456f-bbd7-8105b6e0077d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.54:50052\" within 1s: context deadline exceeded" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.007994 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.008020 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.008609 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:52:54.245992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.008816 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" podUID="8317b14b-20f5-4404-8f4c-bf315ec53736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:52:46.010812 5490 logging.go:55] [core] [Channel #3575 SubChannel #3595]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.54:50052", ServerName: "10.133.0.54:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.54:50052: operation was canceled" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.013780 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" podUID="efd854cd-40a6-4f9f-bd19-cd5d19512dba" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.24:8081/readyz\": dial tcp 10.133.0.24:8081: connect: connection refused" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.015752 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.015794 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" podUID="5123bf1e-0ad6-4406-86b0-2b9a9e025a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.017718 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.017755 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" podUID="dacc99d6-e9da-46e0-841d-14c5f54dd8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.018288 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" podUID="ceb68ad8-3da3-4469-9207-cb473d1dad9e" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.55:50052\" within 1s: context deadline exceeded" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:52:46.018837 5490 logging.go:55] [core] [Channel #3574 SubChannel #3602]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.55:50052", ServerName: "10.133.0.55:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.55:50052: operation was canceled" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.019051 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/odh-model-controller-769875db56-6r4mx" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerName="manager" containerID="cri-o://d619579ca1a11c72ec2427082b17e0d482aeb731037e5af6357ac309ce0af837" gracePeriod=10 Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.021340 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.893785629s: [/var/lib/containers/storage/overlay/e0a03b60981e26ebd3b348305e1cc4f8a6e59f9a86622626a95c1860911d5a38/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.021402 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.893849082s: [/var/lib/containers/storage/overlay/4d471015976bcdbf40a524ee3937cac6973f410e0cd6676ec911929fe86ab450/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.021475 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.893919888s: [/var/lib/containers/storage/overlay/7f9a49b6a8f434f694c6ac7e5a874dcf0fcc2d82b2537cf22433a49f1d6fcd8c/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.021604 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.894038354s: [/var/lib/containers/storage/overlay/86d2e783e7f97e34f5c9a49972a6149e0cb87becc5ea64944ef05785ea0c4e94/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.024380 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" podUID="42e53881-926d-482d-9557-a86d3ae3f9d9" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.98:50052\" within 1s: context deadline exceeded" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:52:46.075751 5490 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c3ed8c6_5871_43dc_aef4_8f440a1060a7.slice/crio-ec00228885dd52018a1ab0bc0d3946855d0149c08a6916c4ed6781db4fbed4ba.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c3ed8c6_5871_43dc_aef4_8f440a1060a7.slice/crio-conmon-ec00228885dd52018a1ab0bc0d3946855d0149c08a6916c4ed6781db4fbed4ba.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.835062 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.32204813s: [/var/lib/containers/storage/overlay/30da62cdff1e4ab9390695abe19c624c429b1e94f5c6dadc6c6e80597a6a3673/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850007 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.837196495s: [/var/lib/containers/storage/overlay/598eadcf073cde1f5997508c644e58c290bca7463312a4a22a30ac40d4551a2d/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_c835066f-39c8-42b4-9f99-1055495bcc69/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850060 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.722481497s: [/var/lib/containers/storage/overlay/766845a055dd70a0d183a1bb82c3066b762d9130cb68c1f36a60372d4c8f20fb/diff /var/log/pods/openshift-insights_insights-runtime-extractor-zngp8_08da34c7-c69a-4eb9-848a-399cab14840e/exporter/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850081 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.7225096s: [/var/lib/containers/storage/overlay/80ea7d5cc4ebc969156c2550b98c3149fcea5fcb9e0fb0cee2dfcc914e3534a2/diff /var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal_fc792e750c2606b177de944a501bf339/haproxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850102 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.722516435s: [/var/lib/containers/storage/overlay/ce1fffd958c87bc28e3747460c780cc088e0059c7dcf38829b819eb1f3b3d4fb/diff /var/log/pods/openshift-monitoring_thanos-querier-b949d9bf6-8b6xc_d395aca5-b8ff-4da1-ad31-dd26e2427c2e/kube-rbac-proxy-rules/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850129 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.722541598s: [/var/lib/containers/storage/overlay/d8f086d2139a3891cbba56d1603c0810b0c6e5663a85eba261af72a01c97649e/diff /var/log/pods/openshift-insights_insights-runtime-extractor-zngp8_08da34c7-c69a-4eb9-848a-399cab14840e/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850154 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.722563721s: [/var/lib/containers/storage/overlay/371a36f63b016c14a7c2f26c4bc48d0d42e9ce17fe27d5f30794ab5a19745dae/diff /var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-968x8_cf66bb31-fee6-49fd-aaea-c8e8d8361c77/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850168 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.71679256s: [/var/lib/containers/storage/overlay/558021b6f3412d66ca76b0c28cb2591d2d4cbbffaf039fd0e9ce3185e070ed36/diff /var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-68drb_55f29950-78b4-4a52-818d-dc235e4631fc/proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850181 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.723548475s: [/var/lib/containers/storage/overlay/f8349c3d17b68d170a51a03145d51b8439276fefa560868232ea68dcbcbb1496/diff /var/log/pods/openshift-dns_node-resolver-z74z2_16b948f8-e3da-4285-8342-27cbd2dbd141/dns-node-resolver/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850198 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.708661241s: [/var/lib/containers/storage/overlay/c81dbbb6bb299b324b4560fb375259db75abcc470c453fdd9239fc9164d79480/diff /var/log/pods/openshift-ingress_e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6_b0596bd0-518a-45ab-8cc1-8ba75268ef6a/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850220 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.708433057s: [/var/lib/containers/storage/overlay/ae391ad936692080268d6af2ce5ce50443a7871160f8b7c81e6d9a54a1e54b4a/diff /var/log/pods/openshift-insights_insights-runtime-extractor-zngp8_08da34c7-c69a-4eb9-848a-399cab14840e/extractor/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850245 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.708454729s: [/var/lib/containers/storage/overlay/fa72440efa1c12e72c07bd93c9190defbe2a7f58e8edd5ed7afed4227335f4c0/diff /var/log/pods/openshift-monitoring_thanos-querier-b949d9bf6-8b6xc_d395aca5-b8ff-4da1-ad31-dd26e2427c2e/prom-label-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850266 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.708471954s: [/var/lib/containers/storage/overlay/401010e57428e36e5c7d63d7650318752412f4168dfc8e13fa5a13baf95fd800/diff /var/log/pods/openshift-monitoring_thanos-querier-b949d9bf6-8b6xc_d395aca5-b8ff-4da1-ad31-dd26e2427c2e/kube-rbac-proxy-metrics/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850292 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.708481806s: [/var/lib/containers/storage/overlay/0cc39e52e51ab73228bbe4fe4c026998b482a0e8a392cd0a3a0453d2bc893006/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rwl_92cf7508-2fc5-49a5-9e71-5d9c73bf200e/ovn-acl-logging/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850312 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.708479798s: [/var/lib/containers/storage/overlay/13025ab3374a87c310ad270b3e805a1493ed8b04c50c8ebaf4da1e34a75d3fbf/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rwl_92cf7508-2fc5-49a5-9e71-5d9c73bf200e/kube-rbac-proxy-node/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850332 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.708494644s: [/var/lib/containers/storage/overlay/f9a3821d5399780420df3d78892361cea3ba30fa4e33a59f570ed7a686e8fb0b/diff /var/log/pods/openshift-monitoring_node-exporter-mtd46_575aa82a-6d21-46c1-80ff-7f756f00c108/node-exporter/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850359 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.708519177s: [/var/lib/containers/storage/overlay/43cb92d8acf491802b08df51f77fbca321aa853c680b42d740223dff4f03cc4d/diff /var/log/pods/openshift-network-diagnostics_network-check-target-4tzpt_d92bdf4f-f9e4-424c-9c87-ec015574ff1a/network-check-target-container/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850376 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.848513493s: [/var/lib/containers/storage/overlay/613bd87eefee210d91d7aec950d2d3e8a116b0f9f80a8a4f2293f4746bce78b9/diff /var/log/pods/openshift-ingress_e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv_d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850381 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.71644467s: [/var/lib/containers/storage/overlay/92b5fc41d5f80c6831e4123b03e27abccc305aca224c7a436718117fe0b760ea/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv_5d9ae28c-045f-456f-bbd7-8105b6e0077d/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850398 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.773236321s: [/var/lib/containers/storage/overlay/55fff06e2f5611dc6c049f66754026723e546c20295d72ed62b77348c3fd8f45/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr_669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0/maas-api/0.log]; will not log again for this container unless duration exceeds 3s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850425 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.708886109s: [/var/lib/containers/storage/overlay/553fe738bd4591523e23b5ada9f939bb6f5aee3d700c51f40149eee2f6cc2fa4/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-t9bq7_6ffcf283-0d79-422e-82be-8b34db814efd/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850447 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.491560634s: [/var/lib/containers/storage/overlay/6715b6c8961630417e435d41b0d73b35e3ea24625998d7e51253f00b04cdb7bc/diff /var/log/pods/openshift-multus_network-metrics-daemon-lkp2n_d680db8a-15e4-4817-bff1-a420f9f54681/network-metrics-daemon/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850474 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.491585865s: [/var/lib/containers/storage/overlay/d4e4dd6027a10d7aec5af3a2f1723b67b873a049f19d863a0d0a3cf855175721/diff /var/log/pods/openshift-dns_dns-default-vcg5q_062e4d3d-4460-4412-93d3-22681a9a74e6/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850495 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.491464709s: [/var/lib/containers/storage/overlay/c95019c90dfe33a9c3ac14c36bd5b1465900e07fcc31b14162953f695c727bb1/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rwl_92cf7508-2fc5-49a5-9e71-5d9c73bf200e/ovnkube-controller/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850520 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.491483534s: [/var/lib/containers/storage/overlay/2d798a8b53f28d62c1dba5ce91e22742c5df7167d6c4a380d97cf4d8ece2ef48/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rwl_92cf7508-2fc5-49a5-9e71-5d9c73bf200e/sbdb/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850542 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.722996645s: [/var/lib/containers/storage/overlay/2b25b385e09e7d7185c8b4c1c7fc0b1b6f66e9bcf47e9197c83c8aff7165424a/diff /var/log/pods/kube-system_global-pull-secret-syncer-gkf5m_7d877fe5-7b94-4645-bff5-6df0796bb637/global-pull-secret-syncer/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850564 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.972861914s: [/var/lib/containers/storage/overlay/bdc10aa02e0596518b6519030f65418c76fe7b720129bde557f773cf89b2110a/diff /var/log/pods/openshift-network-operator_iptables-alerter-frjws_571b679b-fad2-423d-96e5-8d5550bb536d/iptables-alerter/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850591 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.958669687s: [/var/lib/containers/storage/overlay/41b31aa1ff11b186487da0bace0fa061bcfacf4d010fbcf1312fa1bae56dc09d/diff /var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-968x8_cf66bb31-fee6-49fd-aaea-c8e8d8361c77/prometheus-operator/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850610 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.958685871s: [/var/lib/containers/storage/overlay/2ab69660cf08fa7f63a71a0b1ba90adb46948d531213a783d74b10c558b8b21c/diff /var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850635 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.9587036s: [/var/lib/containers/storage/overlay/b03f9ada59adddd8b5d3f95b3ae48aa6cccee6d00f6ceeb18b4a3cdf806d4c7b/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ttmkb_400b676a-a3a0-459d-9da6-8d30b2d3b497/csi-liveness-probe/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850655 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.958721855s: [/var/lib/containers/storage/overlay/f7cc4cad3eeaca4e751a503b5db2741e0df174437a718c4bda77ced2afb78182/diff /var/log/pods/openshift-cluster-node-tuning-operator_tuned-4hwhg_7ad1f3a7-eea4-4104-b3ff-e860da35bf9c/tuned/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850696 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.958710645s: [/var/lib/containers/storage/overlay/d4ffd83f6c6e1752c8bbee86505aaa558ab3bbd206ee8db1cdf3767fc9036320/diff /var/log/pods/openshift-multus_multus-additional-cni-plugins-zcfc2_d7a31b67-a1b9-4b6a-a112-69cd5c833019/kube-multus-additional-cni-plugins/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850729 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.958737303s: [/var/lib/containers/storage/overlay/20d211dc6c712f0952be4d0662405bd52db10b9d51670c2a0ff3993448b61eff/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rwl_92cf7508-2fc5-49a5-9e71-5d9c73bf200e/northd/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850750 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.958736697s: [/var/lib/containers/storage/overlay/0c0e2f2b7059f17901d0f869aa17dd12cb0dfbda050533f0ec365a9a6fae122d/diff /var/log/pods/openshift-ingress-canary_ingress-canary-vh2gt_50818995-4b65-457e-9b55-250b548e80d3/serve-healthcheck-canary/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850774 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.958760091s: [/var/lib/containers/storage/overlay/97c4927847428ac2d46707a4e47dc700b54b7a466ecb9ba2a589c083d2cc64cb/diff /var/log/pods/openshift-multus_network-metrics-daemon-lkp2n_d680db8a-15e4-4817-bff1-a420f9f54681/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850794 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.740725103s: [/var/lib/containers/storage/overlay/f9220bd666687ab2e24d8eb33409b53b70b4654c36e969aab3ddb99e3e3ed371/diff /var/log/pods/openshift-monitoring_thanos-querier-b949d9bf6-8b6xc_d395aca5-b8ff-4da1-ad31-dd26e2427c2e/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850815 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.740556165s: [/var/lib/containers/storage/overlay/4bb713b78934c99bbd28616c3a337ef6b86c60f45a3dd1775ef7a6a54736909b/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rwl_92cf7508-2fc5-49a5-9e71-5d9c73bf200e/kube-rbac-proxy-ovn-metrics/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850840 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.740572127s: [/var/lib/containers/storage/overlay/7220785ff241783a5e3510c9a267df77fe3a4ffcc322086025fec5bcbecca06b/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rwl_92cf7508-2fc5-49a5-9e71-5d9c73bf200e/ovn-controller/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850853 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.716913004s: [/var/lib/containers/storage/overlay/642e7fa3c34800b59531d45475e347034a2aea8e5760b6303f1c859e400b917e/diff /var/log/pods/e2e-models-e2e-worker-w5-bcc7a4_premium-w5-bcc7a4-kserve-864876c669-wlghl_1712a46e-a034-40f6-a276-09a907dd082f/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850872 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.694216033s: [/var/lib/containers/storage/overlay/4324bb324d8dfaffdf271c8ad2eb875d7e658c956c2063a1c90276fd3b3b0ea7/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4_b7c0d839-b3a4-463b-a3a8-3733609d4b89/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850880 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.814715339s: [/var/lib/containers/storage/overlay/9d5e0923a52eda3cbfabac553bb1746858d86b7d3c94119fc7c82e6e3845d816/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl_76c026b4-65c7-4d57-9945-d4ebfb66ca70/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850901 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.546188719s: [/var/lib/containers/storage/overlay/c68b2c3e1b8f3661ef471b7a157b6e323945f8220c96f7b5fcaec66fecd21119/diff /var/log/pods/openshift-ingress_e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5_dacc99d6-e9da-46e0-841d-14c5f54dd8fa/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850902 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.067805448s: [/var/lib/containers/storage/overlay/336903d0a1b8ac056e4cd9d9a82fbc62d7fa1095f79c6656caedfb26b99d9769/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2_ceb68ad8-3da3-4469-9207-cb473d1dad9e/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850922 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.709386282s: [/var/lib/containers/storage/overlay/0137e734be5194a88bfe609cafd14a4d60c0f3dbbdde70d32e06a259c0512d32/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd_288a58cd-9286-422b-bff8-15da47bc1a24/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850932 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.723365866s: [/var/lib/containers/storage/overlay/c7c19d94302b906d82784ecc0108bfae9e27cc21893c27dc7a8929c3d43aa26b/diff /var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-5kbd8_8bae2714-9c1b-44db-a6ef-b0b008cf1fa2/monitoring-plugin/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850948 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.723381862s: [/var/lib/containers/storage/overlay/cbb10c7251409eee4a19a0ecefb1a47190a9804d9c78d0327802d04c8c3b1261/diff /var/log/pods/openshift-monitoring_node-exporter-mtd46_575aa82a-6d21-46c1-80ff-7f756f00c108/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850952 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.608661634s: [/var/lib/containers/storage/overlay/d2e1b5be70964592e83853e6c2ec46cde5315282ab1908aefd6e8def3b608fd1/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_c835066f-39c8-42b4-9f99-1055495bcc69/kube-rbac-proxy-web/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850969 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.709428776s: [/var/lib/containers/storage/overlay/e5e3f60a602f55b84461b3e7b2af58f2dbb1e0107603a08c1a345efdabc2ed73/diff /var/log/pods/openshift-ingress_e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz_db4dcf84-a219-4ff2-b4ff-5549ef2fafe5/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850977 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.303583117s: [/var/lib/containers/storage/overlay/74bc45ed19e2a0d3f62aedc6b7ddc4b3170c77bee2e5941f6fff408d9573c9e2/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_c835066f-39c8-42b4-9f99-1055495bcc69/kube-rbac-proxy-thanos/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850990 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.617089259s: [/var/lib/containers/storage/overlay/27b1eb1691b21d4344b818f77d911d8beae139474632ed3cb0f1bcea46f64ba8/diff /var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-vczbq_e9ecffa6-8de1-421d-952e-a9eeecf8d862/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.850999 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.617097857s: [/var/lib/containers/storage/overlay/f47c79c8b35c4b0ba0751e635bd919c4373e98e56cb06eec726f3fd31bcc0896/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74_e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.851017 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.709491036s: [/var/lib/containers/storage/overlay/d4ff1777ab4b2fd5c4da7d420419184ee8f9946db1ed830ec56007f96640e090/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs_e9f75494-3ba5-460d-a96d-9ab2f4ffa395/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.851021 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.591659783s: [/var/lib/containers/storage/overlay/03853111782d55b217b3835a23309923afe7f2cf1acb078bba3877de7526d408/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw_42e53881-926d-482d-9557-a86d3ae3f9d9/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.851039 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.608746107s: [/var/lib/containers/storage/overlay/e7765c6cbb4168c327b5666672ab58e4f544f86d3f563e10d8a6ffe85650ea9e/diff /var/log/pods/openshift-ingress_e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh_a2dbe7ba-df3f-4b3f-a6c8-6e386172702b/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.851050 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.584053895s: [/var/lib/containers/storage/overlay/9085e23d067d1749ef0653b1083d99f6e0d3c31313aeab20410b43faf236fc4b/diff /var/log/pods/openshift-ingress_e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt_ef5f9523-ec4a-4372-a327-009c9fc40563/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.851066 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.58406728s: [/var/lib/containers/storage/overlay/49a9b673e80022a74f8d15adf0d719d683dfe4e594e11f15c9c6ec409695c012/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz_604d3706-e3e3-4fe2-a9bf-5472a41ee2f0/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.851070 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.183240651s: [/var/lib/containers/storage/overlay/db8d930c5335dd5aae0776fa4fcb8a7673beaa884b1ad8020f39c839fcd04616/diff /var/log/pods/kuadrant-system_authorino-5b8d5bdb45-vldp7_c8be91e5-ce63-477c-91c3-4a24e694a0f3/authorino/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.851086 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.694464769s: [/var/lib/containers/storage/overlay/80d1d207831b2f170c7f3e58262addb57e2a8051868fba6265580575cbb9ccef/diff /var/log/pods/e2e-models-e2e-worker-w3-090ebd_distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw_2b771958-a3d1-40a9-aa74-81b7b33aa1e5/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.851098 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.610113277s: [/var/lib/containers/storage/overlay/bf0f1b311c00800ce04b2c2938b20464fd000c9ebd4f977ac1cb522a6e46a752/diff /var/log/pods/openshift-ingress_e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq_a8a2a69d-e4b7-4338-a7b4-22042b94432f/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.851106 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.33880168s: [/var/lib/containers/storage/overlay/0bb2fc43e1b31cde780b3e374f0382f0c47050f6aadbd98d46317d7acf20e68c/diff /var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm_f64d3e84-440a-4298-9230-ae9e0063c962/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.851118 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.610120286s: [/var/lib/containers/storage/overlay/0f7bdc5e590935e8064384ef01cc7edca930a48944b5c9525863fa4672d1fa54/diff /var/log/pods/e2e-models-e2e-worker-w3-090ebd_distinct-w3-090ebd-kserve-f4dd967c6-tslkr_6c2b41fc-ff94-4a75-affd-81795799528d/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.851139 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.338141595s: [/var/lib/containers/storage/overlay/cf6556de6effaece3531ca67c726ebb368dac8ace8bac17db2866baa2e2552f4/diff /var/log/pods/opendatahub_odh-model-controller-769875db56-6r4mx_c0f0e9f6-2e01-4338-8243-346ffb2c7cc2/manager/1.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.851156 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583396383s: [/var/lib/containers/storage/overlay/2f0db1bec36d893c3ba3ad5e332b274f0fd1018640f785c4833808d7be38d6d5/diff /var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-b78nf_80180652-fb4f-4976-956d-25a47344d581/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.851166 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.694560274s: [/var/lib/containers/storage/overlay/6ca5d3dc094fd0bd9cfa2606b333ca9db05c965371229c21a2ab0726b6feee44/diff /var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rclxh_9e7f6e11-0382-4897-88e3-1d4283f5bed3/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.851182 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.598032325s: [/var/lib/containers/storage/overlay/ef6271e52ddecd379739fd8dae6207156871d5e9dbbcf4cb48646d3d36cf1805/diff /var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8_775a8940-cb22-4774-a43f-c6e7c5364354/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.851200 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.588350706s: [/var/lib/containers/storage/overlay/69ed430e8ff5f9f40546aeb5231a9bb6a5cc844269c0cfd8f78250d5d8c98d7d/diff /var/log/pods/openshift-ingress_e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn_db295e52-5a48-4ec9-a09c-83f8d4b9f62c/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.851204 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.338203861s: [/var/lib/containers/storage/overlay/f84dac45c157c1ba009d07231c7b5e9f7c7fbac1e57224b228c81d39a8ab2c99/diff /var/log/pods/openshift-ingress_e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2_cfa0b7ad-1a4c-4803-8214-e61e9f362694/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.851226 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.338224252s: [/var/lib/containers/storage/overlay/21a35bddc94f868528a63740fe1bd2095d854affb067299e267934c5b60f26aa/diff /var/log/pods/openshift-ingress_kube-auth-proxy-86bb8bf66f-tqrbj_01171a38-57a8-46bb-b2a0-9d69f73f12ea/kube-auth-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.851376 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.584374367s: [/var/lib/containers/storage/overlay/6651b2c5e405c91b727f03dfa221ba361d92d92851f55a12fd9ef5185ce1b331/diff /var/log/pods/openshift-ingress_e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h_dcfb42f0-a328-4179-9a5d-ac6669e132a8/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.852889 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.61898589s: [/var/lib/containers/storage/overlay/d4fc90c3b89a564917caaa941bdfd2a6b37135bcb8ec11782f6aca1380c8db9b/diff /var/log/pods/e2e-models-e2e-worker-w2-f20dc4_premium-w2-f20dc4-kserve-7c8544b957-dz7lc_eca1025f-f016-4b3b-b32f-7ec9c69771b1/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.853125 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.340120902s: [/var/lib/containers/storage/overlay/00927aceb8428d1000af919801f1eac876eadad9351b5d3d770a0cedbe66dae9/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss_da913fbf-0ba8-4607-be4c-5197a22d1a7d/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.853401 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.612411109s: [/var/lib/containers/storage/overlay/03e18c3fd8aa81bba70bcc477de01f70299c15c31b4cdad6322083cb52ca5f6d/diff /var/log/pods/openshift-ingress_e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8_f91c38ef-6298-4315-9a8f-bedd67fcdd1c/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.853546 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.61254849s: [/var/lib/containers/storage/overlay/838b51f1476dc8d7ccbaba95956b4d00eb04f0404d2a6407e2987d65955a5e50/diff /var/log/pods/e2e-models-e2e-worker-w0-d86a77_distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj_3f89ed71-a623-40af-a5b1-5f6333235c14/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.854017 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.620113869s: [/var/lib/containers/storage/overlay/cda436e1f91012f0814b20bc78dd8630d00e9f1a3c15769f41754b41235a4008/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-b-ce9b89-5745885799-w28x7_253719d9-0fac-4d41-bfc3-d4d359ba04fe/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.854042 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.341032776s: [/var/lib/containers/storage/overlay/47ef288497178c53eda71a4daf7201013a10d581f2b9c39c333a8012a6dd9776/diff /var/log/pods/ai-tenant-e2e-inf-a-50b5f2_test-model-50b5f2-kserve-978bd677f-2clcm_7880b1c4-cc27-4692-ae95-5682724c2f4c/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.854117 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.586350127s: [/var/lib/containers/storage/overlay/df7079d1a38b5e69a9826904f2196db879b5931594143fbbdd6fa070097bcdfa/diff /var/log/pods/openshift-ingress_e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7_cdf49e86-d2b1-4888-8475-8a85dd69a6f3/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.854324 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.620415979s: [/var/lib/containers/storage/overlay/e3066158599129fd3724376232856ec4dd6de60e230b7d9254651bfac71ff734/diff /var/log/pods/openshift-ingress_e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8_101e77ad-e8f5-4222-83ee-982ac3954dcc/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.854491 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.727849837s: [/var/lib/containers/storage/overlay/86103ced350fee01fa05300c84fd1e07356b49183f7edde57004f1b77c1eb01f/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ttmkb_400b676a-a3a0-459d-9da6-8d30b2d3b497/csi-node-driver-registrar/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.854519 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.703540701s: [/var/lib/containers/storage/overlay/ba4737b355fce7721f8b2293ef7bd52fbfe361ce0f53fc672a975fc43b978c72/diff /var/log/pods/openshift-ingress_e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87_b0b3f9b0-9f95-4556-87a2-0d7473db75f5/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.870294 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" podUID="775a8940-cb22-4774-a43f-c6e7c5364354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.873784 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.873830 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" podUID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.908568 5490 trace.go:236] Trace[1912717521]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" (31-Oct-2026 03:52:45.745) (total time: 1162ms): Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: Trace[1912717521]: [1.162967485s] [1.162967485s] END Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.910791 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.910853 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" podUID="db295e52-5a48-4ec9-a09c-83f8d4b9f62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.920787 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.920820 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-k9bcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.920841 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" podUID="209cb838-5556-47a9-96ae-e2378ad8c8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.920864 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" podUID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.992861 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.993702 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": context deadline exceeded" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.993663 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.753154707s: [/var/lib/containers/storage/overlay/ffde08c05a5d41f64c4bf3975c18672854b4a6c488c073f2f7be5d826a731713/diff /var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-82ctb_85995108-6b7b-4e56-8260-00b5347d6e6a/postgres/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.993840 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" podUID="55f29950-78b4-4a52-818d-dc235e4631fc" containerName="proxy" probeResult="failure" output="Get \"https://10.133.0.36:8443/health\": context deadline exceeded" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.994945 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.450158536s: [/var/lib/containers/storage/overlay/528a66ac485cac8d0f38eb401f80e38e0baf07bb8cfa8644366732413dd1b2d5/diff /var/log/pods/openshift-ingress_e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2_cb3630f1-9fe3-4a38-8095-2dba83b4e1a7/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.995240 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.995414 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.995513 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.995834 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.995938 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" podUID="f64d3e84-440a-4298-9230-ae9e0063c962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.996023 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" podUID="80180652-fb4f-4976-956d-25a47344d581" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.996301 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.996428 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.996496 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.996537 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.996580 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": context deadline exceeded" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.996591 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:18.750987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.996623 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.996665 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" podUID="80180652-fb4f-4976-956d-25a47344d581" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.996776 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.996781 5490 patch_prober.go:29] interesting pod/thanos-querier-b949d9bf6-8b6xc container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.10:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.996837 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.997429 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.997756 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.998043 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.998268 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.998296 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" podUID="bae51678-b17d-4a9c-98da-77d3ecf0d6ed" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.103:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.998505 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" podUID="e9f75494-3ba5-460d-a96d-9ab2f4ffa395" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.56:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.998614 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" podUID="c9d776b1-fabe-4396-adb1-030d1edad90c" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.48:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.998778 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" podUID="9e7f6e11-0382-4897-88e3-1d4283f5bed3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.49:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.999306 5490 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-5kbd8 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.11:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.999350 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" podUID="8bae2714-9c1b-44db-a6ef-b0b008cf1fa2" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.133.0.11:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:46.999607 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.003685 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.003845 5490 patch_prober.go:29] interesting pod/thanos-querier-b949d9bf6-8b6xc container/kube-rbac-proxy-web namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.133.0.10:9091/-/healthy\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.003854 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/ready\": context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.003881 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/healthy\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.003937 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" podUID="775a8940-cb22-4774-a43f-c6e7c5364354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.004192 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" podUID="55f29950-78b4-4a52-818d-dc235e4631fc" containerName="proxy" probeResult="failure" output="Get \"https://10.133.0.36:8443/ready\": context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.004391 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.008185 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" podUID="5d9ae28c-045f-456f-bbd7-8105b6e0077d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.54:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.013763 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" podUID="f64d3e84-440a-4298-9230-ae9e0063c962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.018459 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" podUID="ceb68ad8-3da3-4469-9207-cb473d1dad9e" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.55:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.019412 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" podUID="4cf50264-0d76-46ca-b2cb-998ae3bbeb1b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.58:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.024544 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" podUID="42e53881-926d-482d-9557-a86d3ae3f9d9" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.98:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.033768 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.033801 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.034798 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.034837 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" podUID="cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.036787 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-6r4mx" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.32:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.036797 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.036861 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.036866 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" podUID="101e77ad-e8f5-4222-83ee-982ac3954dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.036888 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" podUID="dcfb42f0-a328-4179-9a5d-ac6669e132a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.037782 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.037782 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-586ddd746f-nv8gm" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.037782 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.037812 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.037877 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" podUID="f57cc204-16b4-4735-82c1-7cee96c1704d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.038807 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-t9bq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.038844 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" podUID="6ffcf283-0d79-422e-82be-8b34db814efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.039808 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.039842 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" podUID="ca122c09-e327-4371-9879-0870b37db17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:52:47.039870 5490 logging.go:55] [core] [Channel #3591 SubChannel #3593]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.48:50052", ServerName: "10.133.0.48:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:51108->10.133.0.48:50052: use of closed network connection" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:52:47.039870 5490 logging.go:55] [core] [Channel #3589 SubChannel #3592]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.56:50052", ServerName: "10.133.0.56:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:39472->10.133.0.56:50052: use of closed network connection" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:52:47.039885 5490 logging.go:55] [core] [Channel #3584 SubChannel #3587]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.49:50052", ServerName: "10.133.0.49:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:50784->10.133.0.49:50052: use of closed network connection" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.039808 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:52:47.039902 5490 logging.go:55] [core] [Channel #3586 SubChannel #3590]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.103:50052", ServerName: "10.133.0.103:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:59192->10.133.0.103:50052: use of closed network connection" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.039917 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" podUID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.040827 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.040862 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" podUID="cdf49e86-d2b1-4888-8475-8a85dd69a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.044797 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.044816 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" podUID="fd89b915-fe3a-4a8a-aaf5-6927987b2e0d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.22:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.044835 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" podUID="a8a2a69d-e4b7-4338-a7b4-22042b94432f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.044802 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.044866 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" podUID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.044803 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.044908 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" podUID="1ce35023-5f08-4356-8e09-4328952c6040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.045790 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.045827 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" podUID="b0b3f9b0-9f95-4556-87a2-0d7473db75f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.046777 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.046811 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" podUID="db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.048183 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m5.741194479s: [/var/lib/containers/storage/overlay/d98a028022e7e02ceeb779802fe67c47bd275542ce9ca713111c76f7aff10166/diff /var/log/pods/openshift-ingress_e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh_f57cc204-16b4-4735-82c1-7cee96c1704d/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.048759 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.048787 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:52:47.048821 5490 logging.go:55] [core] [Channel #3597 SubChannel #3598]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.54:50052", ServerName: "10.133.0.54:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:47006->10.133.0.54:50052: use of closed network connection" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.049799 5490 patch_prober.go:29] interesting pod/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.049827 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" podUID="015afcb4-d156-4163-a342-f84471c9dd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.049828 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.049871 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" podUID="8317b14b-20f5-4404-8f4c-bf315ec53736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.056788 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.056823 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.057786 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.057820 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" podUID="5123bf1e-0ad6-4406-86b0-2b9a9e025a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.058832 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.058863 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" podUID="dacc99d6-e9da-46e0-841d-14c5f54dd8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:52:47.058871 5490 logging.go:55] [core] [Channel #3603 SubChannel #3604]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.55:50052", ServerName: "10.133.0.55:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:60346->10.133.0.55:50052: use of closed network connection" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:52:47.059808 5490 logging.go:55] [core] [Channel #3606 SubChannel #3607]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.58:50052", ServerName: "10.133.0.58:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:35096->10.133.0.58:50052: use of closed network connection" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:52:47.064833 5490 logging.go:55] [core] [Channel #3610 SubChannel #3611]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.98:50052", ServerName: "10.133.0.98:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:51620->10.133.0.98:50052: use of closed network connection" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.524720 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 38.697852263s: [/var/lib/containers/storage/overlay/577ab036700dd585a07573ad6cb8428437d4ac05a241a6c47a8e6a19107863f6/diff /var/log/pods/cert-manager_cert-manager-webhook-5dbbbf75cc-bdjls_3c3ed8c6-5871-43dc-aef4-8f440a1060a7/cert-manager-webhook/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.871122 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" podUID="775a8940-cb22-4774-a43f-c6e7c5364354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.914814 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.914878 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" podUID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.921352 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-k9bcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.921413 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" podUID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.926197 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.926250 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" podUID="209cb838-5556-47a9-96ae-e2378ad8c8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.952805 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.952861 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" podUID="db295e52-5a48-4ec9-a09c-83f8d4b9f62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.993113 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.994554 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" podUID="55f29950-78b4-4a52-818d-dc235e4631fc" containerName="proxy" probeResult="failure" output="Get \"https://10.133.0.36:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.995628 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.995649 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.995954 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.996044 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" podUID="f64d3e84-440a-4298-9230-ae9e0063c962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.996106 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" podUID="80180652-fb4f-4976-956d-25a47344d581" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.996406 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.996687 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.996868 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.998089 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.998097 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:47.999931 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.004501 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.004516 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.004538 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" podUID="775a8940-cb22-4774-a43f-c6e7c5364354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.009053 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" podUID="5d9ae28c-045f-456f-bbd7-8105b6e0077d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.54:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.014152 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" podUID="f64d3e84-440a-4298-9230-ae9e0063c962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.019526 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" podUID="ceb68ad8-3da3-4469-9207-cb473d1dad9e" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.55:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.019547 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" podUID="4cf50264-0d76-46ca-b2cb-998ae3bbeb1b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.58:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:52:48.050830 5490 logging.go:55] [core] [Channel #3613 SubChannel #3614]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.54:50052", ServerName: "10.133.0.54:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:47016->10.133.0.54:50052: use of closed network connection" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:52:48.059839 5490 logging.go:55] [core] [Channel #3619 SubChannel #3620]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.58:50052", ServerName: "10.133.0.58:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:35112->10.133.0.58:50052: use of closed network connection" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:52:48.059855 5490 logging.go:55] [core] [Channel #3616 SubChannel #3617]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.55:50052", ServerName: "10.133.0.55:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:60350->10.133.0.55:50052: use of closed network connection" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.090846 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.090908 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" podUID="8317b14b-20f5-4404-8f4c-bf315ec53736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.097776 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.097819 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.111030 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.111101 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" podUID="f57cc204-16b4-4735-82c1-7cee96c1704d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.195246 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.229668 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" podUID="c9d776b1-fabe-4396-adb1-030d1edad90c" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.48:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.231744 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" podUID="9e7f6e11-0382-4897-88e3-1d4283f5bed3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.49:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:52:48.271416 5490 logging.go:55] [core] [Channel #3622 SubChannel #3624]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.48:50052", ServerName: "10.133.0.48:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:51124->10.133.0.48:50052: use of closed network connection" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:52:48.272907 5490 logging.go:55] [core] [Channel #3623 SubChannel #3625]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.49:50052", ServerName: "10.133.0.49:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:50806->10.133.0.49:50052: use of closed network connection" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.607720 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" podUID="e9f75494-3ba5-460d-a96d-9ab2f4ffa395" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.56:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:52:48.647286 5490 logging.go:55] [core] [Channel #3628 SubChannel #3629]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.56:50052", ServerName: "10.133.0.56:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:39476->10.133.0.56:50052: use of closed network connection" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.920015 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/healthz\": dial tcp 10.133.0.26:6080: connect: connection refused" Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.922400 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-k9bcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.337411 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.922449 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" podUID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.927426 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.927469 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" podUID="209cb838-5556-47a9-96ae-e2378ad8c8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.968503 5490 trace.go:236] Trace[440494441]: "iptables ChainExists" (31-Oct-2026 03:52:07.112) (total time: 41856ms): Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: Trace[440494441]: [41.856279916s] [41.856279916s] END Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.968554 5490 trace.go:236] Trace[562626333]: "iptables ChainExists" (31-Oct-2026 03:52:06.989) (total time: 41979ms): Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: Trace[562626333]: [41.979166982s] [41.979166982s] END Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.976062 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" podUID="30874c80-e957-4c9a-ab8f-0616de567c3d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.28:8081/readyz\": dial tcp 10.133.0.28:8081: connect: connection refused" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.993867 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" podUID="531c8516-8a62-4add-a9bb-15b02ebb6f5d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.61:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.993887 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" podUID="b7c0d839-b3a4-463b-a3a8-3733609d4b89" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.134:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.993895 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" podUID="da913fbf-0ba8-4607-be4c-5197a22d1a7d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.133:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.996801 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" podUID="669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.57:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.998334 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" podUID="288a58cd-9286-422b-bff8-15da47bc1a24" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.60:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.998671 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" podUID="e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.101:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.998668 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" podUID="604d3706-e3e3-4fe2-a9bf-5472a41ee2f0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.107:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.999004 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" podUID="1303047d-c782-41c7-bc35-d3298c9ef224" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.62:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.999066 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" podUID="c809aea3-9195-4f90-9ffb-ca153af1a5be" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.108:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:48.999422 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" podUID="6f4d14ca-ec36-4daa-8971-93c7dd709b48" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.39:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:49.000775 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" podUID="253719d9-0fac-4d41-bfc3-d4d359ba04fe" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.135:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:49.002097 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" podUID="76c026b4-65c7-4d57-9945-d4ebfb66ca70" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.99:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:49.003112 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" podUID="d2a15588-0492-4ccc-96cc-ab09e1e1402d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.59:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:49.005101 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:49.006452 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" podUID="b8353165-d9b4-445a-9a11-5fdf04307860" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.105:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:49.045793 5490 patch_prober.go:29] interesting pod/dns-default-vcg5q container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:49.045856 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-vcg5q" podUID="062e4d3d-4460-4412-93d3-22681a9a74e6" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:49.397790 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:52:49.516624 5490 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1m11.357s" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:49.690997 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:49.899195 5490 generic.go:356] "Generic (PLEG): container finished" podID="efd854cd-40a6-4f9f-bd19-cd5d19512dba" containerID="a11459fcb77198f0b80d12c2fa87b60c24bbe2c97e259d4af34010cb220aad75" exitCode=1 Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:49.899817 5490 scope.go:122] "RemoveContainer" containerID="a11459fcb77198f0b80d12c2fa87b60c24bbe2c97e259d4af34010cb220aad75" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:50.005748 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:50.378816 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-586ddd746f-nv8gm" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:50.484153 5490 generic.go:356] "Generic (PLEG): container finished" podID="30874c80-e957-4c9a-ab8f-0616de567c3d" containerID="6e71c74b1cffbbe30ddc2c582fd23df1f5369ed32ca47d10dc57c99bf8312729" exitCode=1 Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.006466 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.033778 5490 patch_prober.go:29] interesting pod/dns-default-vcg5q container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.133.0.6:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.033830 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-vcg5q" podUID="062e4d3d-4460-4412-93d3-22681a9a74e6" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.6:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.035795 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.042761 5490 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-t7x7m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.042796 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" podUID="decb2697-571c-48e2-88b3-84a03dfbd80a" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.96:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.043752 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:52:51.161744 5490 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.363s" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.161775 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" event={"ID":"efd854cd-40a6-4f9f-bd19-cd5d19512dba","Type":"ContainerDied","Data":"a11459fcb77198f0b80d12c2fa87b60c24bbe2c97e259d4af34010cb220aad75"} Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.161829 5490 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.161859 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" event={"ID":"30874c80-e957-4c9a-ab8f-0616de567c3d","Type":"ContainerDied","Data":"6e71c74b1cffbbe30ddc2c582fd23df1f5369ed32ca47d10dc57c99bf8312729"} Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.174578 5490 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/maas-controller-586ddd746f-nv8gm" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.178112 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.181699 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.181729 5490 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.181799 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.181952 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.182017 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.182091 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.182117 5490 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.182147 5490 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.182188 5490 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.182261 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-6r4mx" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.182313 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.182345 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.182383 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.182415 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.182467 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.182500 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.185223 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.185275 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.185324 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.185349 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.185373 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.185433 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.185471 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.187552 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.187617 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.187657 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.187711 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.187740 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.187771 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.187800 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.187824 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.187840 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.187859 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.187874 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.187890 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.187906 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.187944 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vcg5q" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.187983 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.188022 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.188075 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.188114 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.188177 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.435563 5490 patch_prober.go:29] interesting pod/thanos-querier-b949d9bf6-8b6xc container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.10:9091/-/ready\": context deadline exceeded" start-of-body= Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:07.469608 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:07.514855 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:07.688279 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:07.733424 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:08.811175 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/health\": context deadline exceeded" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.218565 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.462112 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.465044 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" podUID="6f4d14ca-ec36-4daa-8971-93c7dd709b48" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.39:8443/health\": context deadline exceeded" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.527792 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" podUID="f64d3e84-440a-4298-9230-ae9e0063c962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.529108 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" podUID="775a8940-cb22-4774-a43f-c6e7c5364354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.531154 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" podUID="6f4d14ca-ec36-4daa-8971-93c7dd709b48" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.39:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.546385 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" podUID="253719d9-0fac-4d41-bfc3-d4d359ba04fe" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.135:8443/health\": context deadline exceeded" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.557074 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.559417 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.567944 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" podUID="42e53881-926d-482d-9557-a86d3ae3f9d9" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.98:50052: i/o timeout" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.655367 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.655428 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" podUID="f57cc204-16b4-4735-82c1-7cee96c1704d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.657437 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.657500 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.657529 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" podUID="ca122c09-e327-4371-9879-0870b37db17f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:57.913443 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/ready\": context deadline exceeded" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:57.280230 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" podUID="604d3706-e3e3-4fe2-a9bf-5472a41ee2f0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.107:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:57.954371 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.729652 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.729712 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" podUID="cdf49e86-d2b1-4888-8475-8a85dd69a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.729757 5490 patch_prober.go:29] interesting pod/dns-default-vcg5q container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.729778 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-vcg5q" podUID="062e4d3d-4460-4412-93d3-22681a9a74e6" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:59.176167 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" podUID="b7c0d839-b3a4-463b-a3a8-3733609d4b89" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.134:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.779461 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.779528 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.780789 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.836924 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-586ddd746f-nv8gm" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.837794 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:08.811162 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.884518 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" podUID="101e77ad-e8f5-4222-83ee-982ac3954dcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:02.858143 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.938338 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.938381 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" podUID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.940231 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.144724397s: [/var/lib/containers/storage/overlay/ca5d3ba55b30e9abd4dc8544bbd2089e9a5badf1cb099709e9a9d966079df15f/diff /var/log/pods/openshift-multus_multus-h9wtm_904f42d7-69e3-462f-9055-b6b840800f13/kube-multus/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.940460 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:07.238470 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.942827 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:09.960892 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.657638139s: [/var/lib/containers/storage/overlay/7948638d614a88d109ada86a0a1b6a554c4d74655b83548454c6e101b2ec453e/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.218921 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.462912 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.528625 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" podUID="f64d3e84-440a-4298-9230-ae9e0063c962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.529708 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" podUID="775a8940-cb22-4774-a43f-c6e7c5364354" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.529749 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.532965 5490 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-5kbd8 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.11:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.557839 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.558932 5490 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.558967 5490 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.558981 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.559003 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.559033 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" podUID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.559012 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" podUID="b0b3f9b0-9f95-4556-87a2-0d7473db75f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.560029 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.658231 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-t9bq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.658283 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" podUID="6ffcf283-0d79-422e-82be-8b34db814efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.686463 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" podUID="76c026b4-65c7-4d57-9945-d4ebfb66ca70" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.99:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.701751 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.701811 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.711868 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.712472 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" podUID="4cf50264-0d76-46ca-b2cb-998ae3bbeb1b" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.714122 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.717619 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.727919 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" podUID="80180652-fb4f-4976-956d-25a47344d581" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.733086 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-k9bcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.733128 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" podUID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.734149 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.734186 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" podUID="209cb838-5556-47a9-96ae-e2378ad8c8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.759271 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.759336 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" podUID="db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.838668 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.887311 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.887332 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/ready\": context deadline exceeded" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.887332 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.513955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.887366 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.887387 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.887401 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" podUID="e9ecffa6-8de1-421d-952e-a9eeecf8d862" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": context deadline exceeded" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.887429 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-586ddd746f-nv8gm" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.930871 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" podUID="b8353165-d9b4-445a-9a11-5fdf04307860" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.105:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.936203 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.939313 5490 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.939363 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" podUID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.940697 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.940731 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:10.994399 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" podUID="80180652-fb4f-4976-956d-25a47344d581" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.164596 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" podUID="d2a15588-0492-4ccc-96cc-ab09e1e1402d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.59:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.422600 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.480526252s: [/var/lib/containers/storage/overlay/c7c33d569b225a823d63371821a9c14c690b63ec22b53a144ddeb847403ef871/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.422658 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.480591964s: [/var/lib/containers/storage/overlay/861210e19d556efcedfd1315e4d14b91b7636b89f74b8ec4cfcabb686e0a97c8/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.424578 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" podUID="288a58cd-9286-422b-bff8-15da47bc1a24" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.60:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.424893 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.482846607s: [/var/lib/containers/storage/overlay/d774f9c193fa5423dab0698dfa40f9c8a52280a45a012adc0f3c8b2e3119a7cb/diff /var/log/pods/openshift-monitoring_thanos-querier-b949d9bf6-8b6xc_d395aca5-b8ff-4da1-ad31-dd26e2427c2e/thanos-query/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.424924 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.482936395s: [/var/lib/containers/storage/overlay/4cf5d69f4e2b18bc5db868227301bbfc01f17c7e806a3a1bde7e7ffed91967c7/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_c835066f-39c8-42b4-9f99-1055495bcc69/thanos-sidecar/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.427963 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.485910209s: [/var/lib/containers/storage/overlay/63176ba0fb915a4e6e55059d7057e897de85732ae6b08873fd763a55a7c3469f/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.428009 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.486244198s: [/var/lib/containers/storage/overlay/5eec6f4b57ad92a08c32058ef6a3dc8c458f2bbcec7c76a8553e357bdf3daa3c/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.430644 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" podUID="fd89b915-fe3a-4a8a-aaf5-6927987b2e0d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.22:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.435244 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-insights/insights-runtime-extractor-zngp8" podUID="08da34c7-c69a-4eb9-848a-399cab14840e" containerName="extractor" probeResult="failure" output="command timed out" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.435425 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.435470 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" podUID="db295e52-5a48-4ec9-a09c-83f8d4b9f62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.436115 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="c835066f-39c8-42b4-9f99-1055495bcc69" containerName="prometheus" probeResult="failure" output="command timed out" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.436209 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="c835066f-39c8-42b4-9f99-1055495bcc69" containerName="prometheus" probeResult="failure" output="command timed out" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.436293 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-6r4mx" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.32:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.437356 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" podUID="85995108-6b7b-4e56-8260-00b5347d6e6a" containerName="postgres" probeResult="failure" output="command timed out" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.444657 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.444733 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" podUID="5123bf1e-0ad6-4406-86b0-2b9a9e025a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.446858 5490 patch_prober.go:29] interesting pod/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.446900 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" podUID="015afcb4-d156-4163-a342-f84471c9dd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.447019 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.447044 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" podUID="1ce35023-5f08-4356-8e09-4328952c6040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.459949 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.460039 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.460068 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" podUID="dcfb42f0-a328-4179-9a5d-ac6669e132a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.461297 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.461334 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.462051 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" podUID="e9f75494-3ba5-460d-a96d-9ab2f4ffa395" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.56:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.481768 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.539728579s: [/var/lib/containers/storage/overlay/77246dbf0430366306b09e4a6b47421a8602592d125b973ac00e02b4216dcd09/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.482095 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.540053861s: [/var/lib/containers/storage/overlay/f2bb94e6e397457aa344409b970576ad66aa0bd926aa56af071aa2677da06b80/diff /var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-142-76.us-east-2.compute.internal_9dcc8bbdd4365b42723f25cd49f22a1c/kube-rbac-proxy-crio/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.482502 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.540460978s: [/var/lib/containers/storage/overlay/a00f7772b7b10a65c72d64fa05fe057d56c0afe4751ca58e8c1fa72219f4b27e/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.482658 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.540618851s: [/var/lib/containers/storage/overlay/15f5e5068e0eb6064eeb2c8950e6e308a92742b81a1d6667eb8ce0d62102827f/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.483107 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.490492 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.492339 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.492391 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" podUID="cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.493844 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" podUID="42e53881-926d-482d-9557-a86d3ae3f9d9" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.98:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.523350 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.581350527s: [/var/lib/containers/storage/overlay/a342920f5d50b2f3bbdc71bfa4b1da5605580b44c1506df1bb3be28b0af63902/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.523661 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.581658507s: [/var/lib/containers/storage/overlay/5f5309fb0fc5defc070e8b19e9fd2cf2db7a5d23e87af145070196bd25c9024e/diff /var/log/pods/openshift-monitoring_thanos-querier-b949d9bf6-8b6xc_d395aca5-b8ff-4da1-ad31-dd26e2427c2e/kube-rbac-proxy-web/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.523847 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.581799461s: [/var/lib/containers/storage/overlay/84ee64f6ecba3197d19ca26c984050d2882d7089dda1720ca15e458477b799dc/diff /var/log/pods/openshift-dns_dns-default-vcg5q_062e4d3d-4460-4412-93d3-22681a9a74e6/dns/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.523897 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.581852683s: [/var/lib/containers/storage/overlay/2454321d08a8d89d68537b59787a41eab84984516c21405d1a1d6b3f06d85ee1/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.524005 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.581955595s: [/var/lib/containers/storage/overlay/7fe9e6ce2f875216343a518ef47f07494e0b5877a8cb8e408334bd3f0ec4ec48/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:52:51.435619 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/ready\": context deadline exceeded" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.534060 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.59200731s: [/var/lib/containers/storage/overlay/912f033222c994fe7ce771ab40bdc11ef18069c7e88f154b1f3b39aafb3d217c/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ttmkb_400b676a-a3a0-459d-9da6-8d30b2d3b497/csi-driver/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.542177 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.600187096s: [/var/lib/containers/storage/overlay/3fa8cefef952b4239ba30405e2bed8d785ff4ed6ae1991fbcf35deb31bbf1302/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_c835066f-39c8-42b4-9f99-1055495bcc69/config-reloader/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.544587 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.602593071s: [/var/lib/containers/storage/overlay/192024b7ef573cac42e44b39ca8de424a6889693eaac9bb8e4ddb6bcfd86df31/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.545432 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.603435725s: [/var/lib/containers/storage/overlay/4885800cc02166cfa6060a85b93b6689cdd502c5662a929ad666901a10c24c71/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6rwl_92cf7508-2fc5-49a5-9e71-5d9c73bf200e/nbdb/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.556245 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.614244519s: [/var/lib/containers/storage/overlay/a806db4a7480ba482619ace95bf63481789b5385a232ba349e267139efd9c25f/diff ]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:53:11.603057 5490 logging.go:55] [core] [Channel #3655 SubChannel #3670]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.98:50052", ServerName: "10.133.0.98:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.98:50052: operation was canceled" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.620629 5490 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-t7x7m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.620833 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.620855 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" podUID="8317b14b-20f5-4404-8f4c-bf315ec53736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.624809 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" podUID="8bae2714-9c1b-44db-a6ef-b0b008cf1fa2" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.133.0.11:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.740294 5490 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.854204049s: [/var/lib/containers/storage/overlay/5276f0839df6b9887886ffc5f61fff59f4f85f22ec3fe7e03c0520f9d9d67462/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6_d2a15588-0492-4ccc-96cc-ab09e1e1402d/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.620672 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" podUID="decb2697-571c-48e2-88b3-84a03dfbd80a" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.96:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.765461 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" podUID="669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.57:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:53:11.766010 5490 logging.go:55] [core] [Channel #3652 SubChannel #3656]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.56:50052", ServerName: "10.133.0.56:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.56:50052: operation was canceled" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.768263 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.768507 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.769257 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" podUID="4cf50264-0d76-46ca-b2cb-998ae3bbeb1b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.58:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.772028 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.772635 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:53:11.884981 5490 logging.go:55] [core] [Channel #3658 SubChannel #3659]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.58:50052", ServerName: "10.133.0.58:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:41952->10.133.0.58:50052: use of closed network connection" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.914773 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.915809 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.940797 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.944800 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.945057 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.991853 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.991926 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.991962 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:11.999831 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" podUID="80180652-fb4f-4976-956d-25a47344d581" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:12.048449 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" podUID="80180652-fb4f-4976-956d-25a47344d581" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.102261 5490 patch_prober.go:29] interesting pod/thanos-querier-b949d9bf6-8b6xc container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.10:9091/-/ready\": context deadline exceeded" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.102326 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/ready\": context deadline exceeded" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.102347 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" podUID="604d3706-e3e3-4fe2-a9bf-5472a41ee2f0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.107:8443/health\": context deadline exceeded" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.103515 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" podUID="531c8516-8a62-4add-a9bb-15b02ebb6f5d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.61:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.104089 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" podUID="c809aea3-9195-4f90-9ffb-ca153af1a5be" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.108:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.104246 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.104295 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" podUID="db295e52-5a48-4ec9-a09c-83f8d4b9f62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.104502 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" podUID="6f4d14ca-ec36-4daa-8971-93c7dd709b48" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.39:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.104987 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" podUID="253719d9-0fac-4d41-bfc3-d4d359ba04fe" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.135:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.105101 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.105143 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" podUID="cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.105140 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" podUID="55f29950-78b4-4a52-818d-dc235e4631fc" containerName="proxy" probeResult="failure" output="Get \"https://10.133.0.36:8443/ready\": context deadline exceeded" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.105986 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.106024 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" podUID="1ce35023-5f08-4356-8e09-4328952c6040" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.106540 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" podUID="76c026b4-65c7-4d57-9945-d4ebfb66ca70" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.99:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.109911 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.111829 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-6r4mx" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.32:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.117445 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.117495 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" podUID="dacc99d6-e9da-46e0-841d-14c5f54dd8fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.118285 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" podUID="604d3706-e3e3-4fe2-a9bf-5472a41ee2f0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.107:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.118624 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.118700 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" podUID="dcfb42f0-a328-4179-9a5d-ac6669e132a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.121627 5490 patch_prober.go:29] interesting pod/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.121688 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" podUID="015afcb4-d156-4163-a342-f84471c9dd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.136013 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-586ddd746f-nv8gm" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.136863 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.136900 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" podUID="a8a2a69d-e4b7-4338-a7b4-22042b94432f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.137332 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.137385 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.148665 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.148787 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" podUID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.148827 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" podUID="9e7f6e11-0382-4897-88e3-1d4283f5bed3" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.151025 5490 patch_prober.go:29] interesting pod/dns-default-vcg5q container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.151071 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-vcg5q" podUID="062e4d3d-4460-4412-93d3-22681a9a74e6" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.152029 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" podUID="5d9ae28c-045f-456f-bbd7-8105b6e0077d" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.152547 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" podUID="e9f75494-3ba5-460d-a96d-9ab2f4ffa395" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.179791 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" podUID="fd89b915-fe3a-4a8a-aaf5-6927987b2e0d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.22:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.204357 5490 trace.go:236] Trace[992067674]: "iptables ChainExists" (31-Oct-2026 03:53:09.837) (total time: 3316ms): Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: Trace[992067674]: [3.316118092s] [3.316118092s] END Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.360936 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-6r4mx" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.32:8081/readyz\": dial tcp 10.133.0.32:8081: connect: connection refused" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.414062 5490 trace.go:236] Trace[303213390]: "iptables ChainExists" (31-Oct-2026 03:53:09.762) (total time: 3355ms): Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: Trace[303213390]: [3.355446277s] [3.355446277s] END Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.437982 5490 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"5acd6599b0c4d6104a0d49fefaf92afa22f688ec7ddf1032827396daae057dce"} pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" containerMessage="Container manager failed liveness probe, will be restarted" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.438056 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" containerID="cri-o://5acd6599b0c4d6104a0d49fefaf92afa22f688ec7ddf1032827396daae057dce" gracePeriod=10 Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.437983 5490 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="csi-driver" containerStatusID={"Type":"cri-o","ID":"b682a964bae602e6a2f58ce20a0fb41c736e4b3b99d82dd82bced15005491005"} pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" containerMessage="Container csi-driver failed liveness probe, will be restarted" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.438173 5490 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="csi-node-driver-registrar" containerStatusID={"Type":"cri-o","ID":"28da1a37e8259cbebb74c16558157ef5c27a00697f17b8f4e820d8a31dfbff8b"} pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" containerMessage="Container csi-node-driver-registrar failed liveness probe, will be restarted" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.438248 5490 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"090b98d525cb749c2a0dbb244befddc00cca4ea40d6a6a013699eb4035ae57d8"} pod="opendatahub/maas-controller-586ddd746f-nv8gm" containerMessage="Container manager failed liveness probe, will be restarted" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.438276 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" podUID="400b676a-a3a0-459d-9da6-8d30b2d3b497" containerName="csi-driver" containerID="cri-o://b682a964bae602e6a2f58ce20a0fb41c736e4b3b99d82dd82bced15005491005" gracePeriod=30 Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.438298 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-586ddd746f-nv8gm" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" containerID="cri-o://090b98d525cb749c2a0dbb244befddc00cca4ea40d6a6a013699eb4035ae57d8" gracePeriod=30 Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.438521 5490 scope.go:122] "RemoveContainer" containerID="6e71c74b1cffbbe30ddc2c582fd23df1f5369ed32ca47d10dc57c99bf8312729" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.438884 5490 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="kube-rbac-proxy-web" containerStatusID={"Type":"cri-o","ID":"e1ebe057a3e0ac787165e3f2423426c06641dbcf1d80621a6612e0872ed79dbd"} pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" containerMessage="Container kube-rbac-proxy-web failed liveness probe, will be restarted" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.439164 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" containerID="cri-o://e1ebe057a3e0ac787165e3f2423426c06641dbcf1d80621a6612e0872ed79dbd" gracePeriod=120 Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.444261 5490 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"6e5fed394fb4796510344e30f0eec59f59015e76d311e05c4f6df358cbfd1e25"} pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" containerMessage="Container manager failed liveness probe, will be restarted" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.444332 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" podUID="fd89b915-fe3a-4a8a-aaf5-6927987b2e0d" containerName="manager" containerID="cri-o://6e5fed394fb4796510344e30f0eec59f59015e76d311e05c4f6df358cbfd1e25" gracePeriod=10 Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.445045 5490 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="proxy" containerStatusID={"Type":"cri-o","ID":"20c1a8fe58f1b857d42ccc0528bd0a4c43f1aa5bbcec5f42f3409e3ac599573f"} pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" containerMessage="Container proxy failed liveness probe, will be restarted" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.445105 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" podUID="55f29950-78b4-4a52-818d-dc235e4631fc" containerName="proxy" containerID="cri-o://20c1a8fe58f1b857d42ccc0528bd0a4c43f1aa5bbcec5f42f3409e3ac599573f" gracePeriod=30 Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.793918 5490 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.793953 5490 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.793981 5490 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 03:53:40.827304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.794001 5490 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-76.us-east-2.compute.internal" event="NodeReady" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:13.964763 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/healthz\": dial tcp 10.133.0.26:6080: connect: connection refused" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:14.147559 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" podUID="55f29950-78b4-4a52-818d-dc235e4631fc" containerName="proxy" probeResult="failure" output="Get \"https://10.133.0.36:8443/ready\": context deadline exceeded" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:14.423856 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:14.423901 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" podUID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:14.710490 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:14.710534 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" podUID="cdf49e86-d2b1-4888-8475-8a85dd69a6f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:14.710714 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:14.710736 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:14.781820 5490 patch_prober.go:29] interesting pod/dns-default-vcg5q container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.133.0.6:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:14.781878 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-vcg5q" podUID="062e4d3d-4460-4412-93d3-22681a9a74e6" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.6:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:14.868074 5490 trace.go:236] Trace[1214152816]: "Calculate volume metrics of metrics-client-ca for pod openshift-monitoring/node-exporter-mtd46" (31-Oct-2026 03:53:13.634) (total time: 1233ms): Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: Trace[1214152816]: [1.233091929s] [1.233091929s] END Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:53:15.272121 5490 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="5.337s" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:16.525481 5490 patch_prober.go:29] interesting pod/thanos-querier-b949d9bf6-8b6xc container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.10:9091/-/ready\": context deadline exceeded" start-of-body= Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:16.525557 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/ready\": context deadline exceeded" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:17.526565 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:17.527649 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:17.815573 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:17.815587 5490 patch_prober.go:29] interesting pod/dns-default-vcg5q container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:17.815793 5490 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-t7x7m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:17.815865 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-vcg5q" podUID="062e4d3d-4460-4412-93d3-22681a9a74e6" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:17.815865 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" podUID="decb2697-571c-48e2-88b3-84a03dfbd80a" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.96:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:18.315399 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:18.315472 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" podUID="209cb838-5556-47a9-96ae-e2378ad8c8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:18.527734 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:18.527866 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:18.919844 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/healthz\": dial tcp 10.133.0.26:6080: connect: connection refused" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:21.023970 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" podUID="288a58cd-9286-422b-bff8-15da47bc1a24" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.60:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:21.025198 5490 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-5kbd8 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.11:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:21.025264 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" podUID="8bae2714-9c1b-44db-a6ef-b0b008cf1fa2" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.133.0.11:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:22.988906 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" podUID="c809aea3-9195-4f90-9ffb-ca153af1a5be" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.108:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:26.672810 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:28.886358 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" podUID="253719d9-0fac-4d41-bfc3-d4d359ba04fe" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.135:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:28.904975 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:31.003733 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:31.914314 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" podUID="76c026b4-65c7-4d57-9945-d4ebfb66ca70" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.99:8443/health\": context deadline exceeded" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:32.063318 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" podUID="253719d9-0fac-4d41-bfc3-d4d359ba04fe" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.135:8443/health\": context deadline exceeded" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:33.160807 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" podUID="da913fbf-0ba8-4607-be4c-5197a22d1a7d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.133:8443/health\": context deadline exceeded" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:33.799431 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" podUID="288a58cd-9286-422b-bff8-15da47bc1a24" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.60:8443/health\": context deadline exceeded" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:33.912437 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" podUID="b7c0d839-b3a4-463b-a3a8-3733609d4b89" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.134:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:33.927383 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" podUID="85995108-6b7b-4e56-8260-00b5347d6e6a" containerName="postgres" probeResult="failure" output="command timed out" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:33.927972 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:33.687643 5490 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.015672 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" podUID="db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:33.799288 5490 patch_prober.go:29] interesting pod/thanos-querier-b949d9bf6-8b6xc container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.10:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.016113 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:33.927915 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-t9bq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.022976 5490 patch_prober.go:29] interesting pod/dns-default-vcg5q container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.023022 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-vcg5q" podUID="062e4d3d-4460-4412-93d3-22681a9a74e6" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.6:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.112228 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" podUID="6ffcf283-0d79-422e-82be-8b34db814efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.130874 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.482316 5490 trace.go:236] Trace[504667878]: "Calculate volume metrics of data for pod odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" (31-Oct-2026 03:53:13.434) (total time: 21047ms): Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: Trace[504667878]: [21.047509702s] [21.047509702s] END Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.685347 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.754612 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-6r4mx" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.32:8081/readyz\": dial tcp 10.133.0.32:8081: connect: connection refused" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.755200 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/odh-model-controller-769875db56-6r4mx" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.32:8081/readyz\": dial tcp 10.133.0.32:8081: connect: connection refused" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.757251 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.797069 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" podUID="4cf50264-0d76-46ca-b2cb-998ae3bbeb1b" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.58:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:53:34.798948 5490 logging.go:55] [core] [Channel #3697 SubChannel #3707]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.58:50052", ServerName: "10.133.0.58:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.58:50052: operation was canceled" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.799562 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.802304 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" podUID="42e53881-926d-482d-9557-a86d3ae3f9d9" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.98:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:53:34.802768 5490 logging.go:55] [core] [Channel #3695 SubChannel #3696]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.98:50052", ServerName: "10.133.0.98:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.98:50052: operation was canceled" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.809499 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/healthz\": dial tcp 10.133.0.26:6080: connect: connection refused" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.812020 5490 ???:1] "http: TLS handshake error from 10.133.0.13:55224: read tcp 10.0.142.76:10250->10.133.0.13:55224: read: connection reset by peer" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.812363 5490 patch_prober.go:29] interesting pod/thanos-querier-b949d9bf6-8b6xc container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.10:9091/-/ready\": dial tcp 10.133.0.10:9091: connect: connection refused" start-of-body= Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.812419 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/ready\": dial tcp 10.133.0.10:9091: connect: connection refused" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.824773 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="c835066f-39c8-42b4-9f99-1055495bcc69" containerName="prometheus" probeResult="failure" output="command timed out" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.824846 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="c835066f-39c8-42b4-9f99-1055495bcc69" containerName="prometheus" probeResult="failure" output="command timed out" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.825133 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-insights/insights-runtime-extractor-zngp8" podUID="08da34c7-c69a-4eb9-848a-399cab14840e" containerName="extractor" probeResult="failure" output="command timed out" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:53:34.836846 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="86d75c08ecd5493020526daadaf8a8ee9ee5987fb034c16e3a91392ef7ed9ada" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.844374 5490 patch_prober.go:29] interesting pod/dns-default-vcg5q container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.133.0.6:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.844422 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-vcg5q" podUID="062e4d3d-4460-4412-93d3-22681a9a74e6" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.6:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.845235 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/readyz\": dial tcp 10.133.0.31:8081: connect: connection refused" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.846759 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" podUID="fd89b915-fe3a-4a8a-aaf5-6927987b2e0d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.22:8081/readyz\": dial tcp 10.133.0.22:8081: connect: connection refused" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.847001 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" podUID="fd89b915-fe3a-4a8a-aaf5-6927987b2e0d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.22:8081/readyz\": dial tcp 10.133.0.22:8081: connect: connection refused" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:53:34.890971 5490 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd395aca5_b8ff_4da1_ad31_dd26e2427c2e.slice/crio-e1ebe057a3e0ac787165e3f2423426c06641dbcf1d80621a6612e0872ed79dbd.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:53:34.891006 5490 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd395aca5_b8ff_4da1_ad31_dd26e2427c2e.slice/crio-e1ebe057a3e0ac787165e3f2423426c06641dbcf1d80621a6612e0872ed79dbd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce529bc9_a44d_4808_8d2c_996e14f88a54.slice/crio-conmon-5acd6599b0c4d6104a0d49fefaf92afa22f688ec7ddf1032827396daae057dce.scope\": RecentStats: unable to find data in memory cache]" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.912901 5490 patch_prober.go:29] interesting pod/monitoring-plugin-77f4f76594-5kbd8 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.11:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.912955 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5kbd8" podUID="8bae2714-9c1b-44db-a6ef-b0b008cf1fa2" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.133.0.11:9443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.914656 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" podUID="76c026b4-65c7-4d57-9945-d4ebfb66ca70" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.99:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.953809 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:34.953855 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" podUID="209cb838-5556-47a9-96ae-e2378ad8c8b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.229224 5490 generic.go:356] "Generic (PLEG): container finished" podID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerID="ec00228885dd52018a1ab0bc0d3946855d0149c08a6916c4ed6781db4fbed4ba" exitCode=124 Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:53:35.386588 5490 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="20.114s" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.386772 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.387571 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.401637 5490 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.401715 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" event={"ID":"3c3ed8c6-5871-43dc-aef4-8f440a1060a7","Type":"ContainerDied","Data":"ec00228885dd52018a1ab0bc0d3946855d0149c08a6916c4ed6781db4fbed4ba"} Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.401807 5490 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.401859 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.401877 5490 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-insights/insights-runtime-extractor-zngp8" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.401934 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.402026 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.402152 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.402214 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.402256 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.402301 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.402352 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.402396 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.402478 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.402731 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.402829 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.402885 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.402911 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.402961 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.403015 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.403198 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.403405 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.403540 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-vczbq" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.403608 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.403704 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.403881 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.403997 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.404328 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.404481 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.404539 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.404601 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.404644 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.404689 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.404721 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.404766 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.404815 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.404865 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.404954 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.405004 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.405072 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.405314 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.405404 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.405483 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.405550 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.405609 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.405719 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.405800 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.405823 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh"] Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.405944 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.406018 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.406146 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.406252 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vcg5q" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.406320 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.406373 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw"] Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.406409 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p"] Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.406457 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt"] Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.406487 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h"] Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.410763 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.413103 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.413214 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.422548 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-b78nf" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.423815 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.428117 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.428256 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.429033 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.430151 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.432931 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.435984 5490 patch_prober.go:29] interesting pod/thanos-querier-b949d9bf6-8b6xc container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.10:9091/-/ready\": dial tcp 10.133.0.10:9091: connect: connection refused" start-of-body= Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.436054 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/ready\": dial tcp 10.133.0.10:9091: connect: connection refused" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.443219 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.744013 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" podUID="55f29950-78b4-4a52-818d-dc235e4631fc" containerName="proxy" probeResult="failure" output="Get \"https://10.133.0.36:8443/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.744044 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" podUID="ceb68ad8-3da3-4469-9207-cb473d1dad9e" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.55:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.744033 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" podUID="e9f75494-3ba5-460d-a96d-9ab2f4ffa395" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.56:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.744128 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.744031 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gzjbq" podUID="c9d776b1-fabe-4396-adb1-030d1edad90c" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.48:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.744180 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:53:35.784849 5490 logging.go:55] [core] [Channel #3699 SubChannel #3701]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.55:50052", ServerName: "10.133.0.55:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:46562->10.133.0.55:50052: use of closed network connection" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:53:35.784870 5490 logging.go:55] [core] [Channel #3700 SubChannel #3703]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.56:50052", ServerName: "10.133.0.56:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:46882->10.133.0.56:50052: use of closed network connection" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:53:35.784870 5490 logging.go:55] [core] [Channel #3698 SubChannel #3702]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.48:50052", ServerName: "10.133.0.48:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:60978->10.133.0.48:50052: use of closed network connection" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.794239 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv"] Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.802538 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" podUID="42e53881-926d-482d-9557-a86d3ae3f9d9" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.98:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.817764 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" podUID="5d9ae28c-045f-456f-bbd7-8105b6e0077d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.54:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.836123 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" podUID="bae51678-b17d-4a9c-98da-77d3ecf0d6ed" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.103:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.840345 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-55fbbbf9dc-rclxh" podUID="9e7f6e11-0382-4897-88e3-1d4283f5bed3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.49:50052\" within 1s: context deadline exceeded" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:53:35.843845 5490 logging.go:55] [core] [Channel #3708 SubChannel #3709]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.98:50052", ServerName: "10.133.0.98:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:33706->10.133.0.98:50052: use of closed network connection" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:35.852801 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-586ddd746f-nv8gm" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:53:35.859834 5490 logging.go:55] [core] [Channel #3711 SubChannel #3712]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.54:50052", ServerName: "10.133.0.54:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:37104->10.133.0.54:50052: use of closed network connection" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:53:35.876872 5490 logging.go:55] [core] [Channel #3714 SubChannel #3715]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.103:50052", ServerName: "10.133.0.103:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:44928->10.133.0.103:50052: use of closed network connection" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:53:35.880860 5490 logging.go:55] [core] [Channel #3717 SubChannel #3718]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.49:50052", ServerName: "10.133.0.49:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:38180->10.133.0.49:50052: use of closed network connection" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.161107 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" podUID="da913fbf-0ba8-4607-be4c-5197a22d1a7d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.133:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:40.951357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.161144 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" podUID="e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.101:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.261255 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" podUID="c809aea3-9195-4f90-9ffb-ca153af1a5be" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.108:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.393414 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.400125 5490 generic.go:356] "Generic (PLEG): container finished" podID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerID="d619579ca1a11c72ec2427082b17e0d482aeb731037e5af6357ac309ce0af837" exitCode=137 Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.400197 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-6r4mx" event={"ID":"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2","Type":"ContainerDied","Data":"d619579ca1a11c72ec2427082b17e0d482aeb731037e5af6357ac309ce0af837"} Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.400265 5490 scope.go:122] "RemoveContainer" containerID="7209ae60ec915af7c0e7e97853a9c90448c6f22efc9001bbe9ec2bededab41a1" Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.440181 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.475572 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ttmkb_400b676a-a3a0-459d-9da6-8d30b2d3b497/csi-driver/0.log" Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.475626 5490 generic.go:356] "Generic (PLEG): container finished" podID="400b676a-a3a0-459d-9da6-8d30b2d3b497" containerID="b682a964bae602e6a2f58ce20a0fb41c736e4b3b99d82dd82bced15005491005" exitCode=2 Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.475840 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" event={"ID":"400b676a-a3a0-459d-9da6-8d30b2d3b497","Type":"ContainerDied","Data":"b682a964bae602e6a2f58ce20a0fb41c736e4b3b99d82dd82bced15005491005"} Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.476033 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" podUID="5123bf1e-0ad6-4406-86b0-2b9a9e025a45" containerName="istio-proxy" containerID="cri-o://744a3ca1bfa7c5a9395b7d6b7b5d8b367ba6b48b2d8954fd459ae579b3c71f14" gracePeriod=30 Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.505280 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" containerID="cri-o://e80da2fd7fb9cb92560ec74ab6aa1a423731a1c2640319fd6b6fdb73d4762e62" gracePeriod=30 Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.510827 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.510909 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" podUID="dcfb42f0-a328-4179-9a5d-ac6669e132a8" containerName="istio-proxy" containerID="cri-o://2a9717e9f16ee89232fadcc80299291331ed5ad5d1c8a5263458de8973929d4a" gracePeriod=30 Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.512425 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.531637 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" podUID="85995108-6b7b-4e56-8260-00b5347d6e6a" containerName="postgres" probeResult="failure" output="command timed out" Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.600980 5490 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="extractor" containerStatusID={"Type":"cri-o","ID":"cfcc76b08afa1eb87d3dedfa1ab09840a6be4e413e9ac4758b279564e30f5bd5"} pod="openshift-insights/insights-runtime-extractor-zngp8" containerMessage="Container extractor failed liveness probe, will be restarted" Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.601048 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/insights-runtime-extractor-zngp8" podUID="08da34c7-c69a-4eb9-848a-399cab14840e" containerName="extractor" containerID="cri-o://cfcc76b08afa1eb87d3dedfa1ab09840a6be4e413e9ac4758b279564e30f5bd5" gracePeriod=30 Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.808448 5490 plugin_watcher.go:214] "Removing socket path from desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.878781 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" podUID="bae51678-b17d-4a9c-98da-77d3ecf0d6ed" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:36.951436 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" podUID="400b676a-a3a0-459d-9da6-8d30b2d3b497" containerName="csi-node-driver-registrar" containerID="cri-o://28da1a37e8259cbebb74c16558157ef5c27a00697f17b8f4e820d8a31dfbff8b" gracePeriod=28 Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:37.539608 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerName="istio-proxy" containerID="cri-o://e24242cc752af656810ca6373322bafdec557f3a573129be7802d1166a376163" gracePeriod=30 Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:37.541232 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" podUID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerName="istio-proxy" containerID="cri-o://8212151b6af29b681d3346cd5171621f62899608c3f910e235e612a058935331" gracePeriod=30 Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:37.542729 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" podUID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" containerName="istio-proxy" containerID="cri-o://042b2b5699dc8a0e8f1b9b4fc6fa574f84cbd93097f524eac8d7af8f87354bb6" gracePeriod=30 Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:37.758024 5490 csi_plugin.go:271] kubernetes.io/csi: registrationHandler.DeRegisterPlugin request for plugin ebs.csi.aws.com, endpoint /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:37.757946 5490 reconciler.go:150] "OperationExecutor.UnregisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"0001-01-01T00:00:00Z","UUID":"b3b95c3e-b69e-42c7-9e56-30754a34a248","Handler":{},"Name":"ebs.csi.aws.com","Endpoint":"/var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock"} Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:37.928779 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8"] Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:37.928985 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" podUID="101e77ad-e8f5-4222-83ee-982ac3954dcc" containerName="istio-proxy" containerID="cri-o://64382e4e17f65f1d673c6d45af5f19a58840ec3a10e32b2caf5262d45b8430c2" gracePeriod=30 Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:38.260154 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/readyz\": dial tcp 10.133.0.31:8081: connect: connection refused" Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:38.283833 5490 trace.go:236] Trace[1155111513]: "Calculate volume metrics of manifests for pod opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" (31-Oct-2026 03:52:47.217) (total time: 51066ms): Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: Trace[1155111513]: [51.0663108s] [51.0663108s] END Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:38.527473 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" podUID="85995108-6b7b-4e56-8260-00b5347d6e6a" containerName="postgres" probeResult="failure" output="command timed out" Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:38.527475 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="c835066f-39c8-42b4-9f99-1055495bcc69" containerName="prometheus" probeResult="failure" output="command timed out" Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:38.527474 5490 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="c835066f-39c8-42b4-9f99-1055495bcc69" containerName="prometheus" probeResult="failure" output="command timed out" Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:38.550777 5490 generic.go:356] "Generic (PLEG): container finished" podID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerID="e1ebe057a3e0ac787165e3f2423426c06641dbcf1d80621a6612e0872ed79dbd" exitCode=0 Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:38.550848 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" event={"ID":"d395aca5-b8ff-4da1-ad31-dd26e2427c2e","Type":"ContainerDied","Data":"e1ebe057a3e0ac787165e3f2423426c06641dbcf1d80621a6612e0872ed79dbd"} Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:38.553453 5490 generic.go:356] "Generic (PLEG): container finished" podID="fd89b915-fe3a-4a8a-aaf5-6927987b2e0d" containerID="6e5fed394fb4796510344e30f0eec59f59015e76d311e05c4f6df358cbfd1e25" exitCode=137 Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:38.553528 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" event={"ID":"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d","Type":"ContainerDied","Data":"6e5fed394fb4796510344e30f0eec59f59015e76d311e05c4f6df358cbfd1e25"} Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:38.555561 5490 generic.go:356] "Generic (PLEG): container finished" podID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerID="5acd6599b0c4d6104a0d49fefaf92afa22f688ec7ddf1032827396daae057dce" exitCode=137 Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:38.555632 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" event={"ID":"ce529bc9-a44d-4808-8d2c-996e14f88a54","Type":"ContainerDied","Data":"5acd6599b0c4d6104a0d49fefaf92afa22f688ec7ddf1032827396daae057dce"} Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:38.557954 5490 generic.go:356] "Generic (PLEG): container finished" podID="08da34c7-c69a-4eb9-848a-399cab14840e" containerID="cfcc76b08afa1eb87d3dedfa1ab09840a6be4e413e9ac4758b279564e30f5bd5" exitCode=143 Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:38.558025 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zngp8" event={"ID":"08da34c7-c69a-4eb9-848a-399cab14840e","Type":"ContainerDied","Data":"cfcc76b08afa1eb87d3dedfa1ab09840a6be4e413e9ac4758b279564e30f5bd5"} Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:38.822132 5490 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:38.920244 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" podUID="3c3ed8c6-5871-43dc-aef4-8f440a1060a7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.26:6080/healthz\": dial tcp 10.133.0.26:6080: connect: connection refused" Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:39.471432 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:40.106457 5490 trace.go:236] Trace[1511800045]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" (31-Oct-2026 03:52:45.220) (total time: 54885ms): Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: Trace[1511800045]: [54.885920627s] [54.885920627s] END Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:40.177130 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:40.324534 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:40.435530 5490 patch_prober.go:29] interesting pod/thanos-querier-b949d9bf6-8b6xc container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.10:9091/-/ready\": dial tcp 10.133.0.10:9091: connect: connection refused" start-of-body= Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:40.435594 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" podUID="d395aca5-b8ff-4da1-ad31-dd26e2427c2e" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.133.0.10:9091/-/ready\": dial tcp 10.133.0.10:9091: connect: connection refused" Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:40.799954 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl"] Oct 31 03:53:41.151762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:40.805207 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd"] Oct 31 03:53:41.242871 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.177359 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-586ddd746f-nv8gm" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": dial tcp 10.133.0.35:8081: connect: connection refused" Oct 31 03:53:41.242871 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.185211 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" podUID="fd89b915-fe3a-4a8a-aaf5-6927987b2e0d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.22:8081/readyz\": dial tcp 10.133.0.22:8081: connect: connection refused" Oct 31 03:53:41.245415 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.243834 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:53:41.599386 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.599351 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ttmkb_400b676a-a3a0-459d-9da6-8d30b2d3b497/csi-driver/0.log" Oct 31 03:53:41.599942 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.599900 5490 generic.go:356] "Generic (PLEG): container finished" podID="400b676a-a3a0-459d-9da6-8d30b2d3b497" containerID="28da1a37e8259cbebb74c16558157ef5c27a00697f17b8f4e820d8a31dfbff8b" exitCode=0 Oct 31 03:53:41.606414 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.606382 5490 generic.go:356] "Generic (PLEG): container finished" podID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerID="090b98d525cb749c2a0dbb244befddc00cca4ea40d6a6a013699eb4035ae57d8" exitCode=0 Oct 31 03:53:41.606808 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.606777 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" event={"ID":"400b676a-a3a0-459d-9da6-8d30b2d3b497","Type":"ContainerDied","Data":"28da1a37e8259cbebb74c16558157ef5c27a00697f17b8f4e820d8a31dfbff8b"} Oct 31 03:53:41.606910 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.606848 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586ddd746f-nv8gm" event={"ID":"2238dc08-3a3e-426e-a8f7-22bec5f016c8","Type":"ContainerDied","Data":"090b98d525cb749c2a0dbb244befddc00cca4ea40d6a6a013699eb4035ae57d8"} Oct 31 03:53:41.640999 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:53:41.640954 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824618ba_ce21_4eeb_bf27_f1a861327863.slice/crio-4b56be18a650aa009333128d27cdf579fcced1a750d5b44f2f8ee98f969feefa WatchSource:0}: Error finding container 4b56be18a650aa009333128d27cdf579fcced1a750d5b44f2f8ee98f969feefa: Status 404 returned error can't find the container with id 4b56be18a650aa009333128d27cdf579fcced1a750d5b44f2f8ee98f969feefa Oct 31 03:53:41.661149 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:53:41.661116 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05bbcb81_55ce_4387_89c3_f6350fe8636f.slice/crio-21a8842aaace157244dd9e7a44d2ba6b0f18415fe58d4829ceea2fb2fad0bd08 WatchSource:0}: Error finding container 21a8842aaace157244dd9e7a44d2ba6b0f18415fe58d4829ceea2fb2fad0bd08: Status 404 returned error can't find the container with id 21a8842aaace157244dd9e7a44d2ba6b0f18415fe58d4829ceea2fb2fad0bd08 Oct 31 03:53:41.672760 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.672539 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-6r4mx" Oct 31 03:53:41.680920 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.680876 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:53:41.681080 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.681006 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:53:41.681080 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.681057 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:53:41.684117 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.684081 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:53:41.684224 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.684168 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:53:41.684308 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.684219 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:53:41.716717 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.716655 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-82ctb" Oct 31 03:53:41.717226 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.717195 5490 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:53:41.717357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.717262 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" podUID="dcfb42f0-a328-4179-9a5d-ac6669e132a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:53:41.794976 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.794944 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-cert\" (UniqueName: \"kubernetes.io/secret/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-cert\") pod \"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2\" (UID: \"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2\") " Oct 31 03:53:41.795182 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.795129 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-kube-api-access-n87jc\" (UniqueName: \"kubernetes.io/projected/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-kube-api-access-n87jc\") pod \"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2\" (UID: \"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2\") " Oct 31 03:53:41.795250 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:41.795178 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-metrics-certs\") pod \"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2\" (UID: \"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2\") " Oct 31 03:53:42.703150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:42.617130 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" event={"ID":"05bbcb81-55ce-4387-89c3-f6350fe8636f","Type":"ContainerStarted","Data":"21a8842aaace157244dd9e7a44d2ba6b0f18415fe58d4829ceea2fb2fad0bd08"} Oct 31 03:53:42.703150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:42.621990 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" event={"ID":"d395aca5-b8ff-4da1-ad31-dd26e2427c2e","Type":"ContainerStarted","Data":"2ef3f3d48ed9563407862ff36e12d421fb365585083805506b09052ccc817b2a"} Oct 31 03:53:42.703150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:42.622106 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:53:42.703150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:42.623379 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-6r4mx" Oct 31 03:53:42.703150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:42.623401 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-6r4mx" event={"ID":"c0f0e9f6-2e01-4338-8243-346ffb2c7cc2","Type":"ContainerDied","Data":"5849752547eed196e235e4809d34f2d225728b395f98ec7bba9d29a9814af3c0"} Oct 31 03:53:42.703150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:42.623465 5490 scope.go:122] "RemoveContainer" containerID="d619579ca1a11c72ec2427082b17e0d482aeb731037e5af6357ac309ce0af837" Oct 31 03:53:42.703150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:42.625709 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" event={"ID":"efd854cd-40a6-4f9f-bd19-cd5d19512dba","Type":"ContainerStarted","Data":"d79ed8df8ca57550951245f437acd93412f0f556844f05960ee43f008753cd56"} Oct 31 03:53:42.703150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:42.625778 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" Oct 31 03:53:42.703150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:42.627024 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" event={"ID":"824618ba-ce21-4eeb-bf27-f1a861327863","Type":"ContainerStarted","Data":"4b56be18a650aa009333128d27cdf579fcced1a750d5b44f2f8ee98f969feefa"} Oct 31 03:53:42.703150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:42.628937 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" event={"ID":"30874c80-e957-4c9a-ab8f-0616de567c3d","Type":"ContainerStarted","Data":"da65448200a5c2380a1c4af37d4869c1637f5304b00f61d1c536160566c9826a"} Oct 31 03:53:42.703150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:42.630785 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" event={"ID":"3c3ed8c6-5871-43dc-aef4-8f440a1060a7","Type":"ContainerStarted","Data":"e5dbaf246ca5a59a6b30b9bb0293c742f13950c1b03a128497146dbe9d490917"} Oct 31 03:53:42.703150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:42.630871 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" Oct 31 03:53:43.053586 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.053471 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-cert" pod "c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" (UID: "c0f0e9f6-2e01-4338-8243-346ffb2c7cc2"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:53:43.053586 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.053566 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-kube-api-access-n87jc" pod "c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" (UID: "c0f0e9f6-2e01-4338-8243-346ffb2c7cc2"). InnerVolumeSpecName "kube-api-access-n87jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:43.118385 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.118350 5490 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:43.118385 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.118385 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n87jc\" (UniqueName: \"kubernetes.io/projected/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-kube-api-access-n87jc\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:43.145597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.145550 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-metrics-certs" pod "c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" (UID: "c0f0e9f6-2e01-4338-8243-346ffb2c7cc2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:53:43.219986 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.219821 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:43.260200 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.260163 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" podUID="ce529bc9-a44d-4808-8d2c-996e14f88a54" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/readyz\": dial tcp 10.133.0.31:8081: connect: connection refused" Oct 31 03:53:43.344580 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.344529 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/odh-model-controller-769875db56-6r4mx"] Oct 31 03:53:43.350212 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.350182 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/odh-model-controller-769875db56-6r4mx"] Oct 31 03:53:43.597802 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.581207 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" path="/var/lib/kubelet/pods/c0f0e9f6-2e01-4338-8243-346ffb2c7cc2/volumes" Oct 31 03:53:43.637038 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.631668 5490 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 03:53:43.637038 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.632042 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" podUID="5123bf1e-0ad6-4406-86b0-2b9a9e025a45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 03:53:43.637038 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.632732 5490 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 31 03:53:43.637038 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.632784 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 31 03:53:43.647247 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.647106 5490 generic.go:356] "Generic (PLEG): container finished" podID="dcfb42f0-a328-4179-9a5d-ac6669e132a8" containerID="2a9717e9f16ee89232fadcc80299291331ed5ad5d1c8a5263458de8973929d4a" exitCode=0 Oct 31 03:53:43.647247 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.647200 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" event={"ID":"dcfb42f0-a328-4179-9a5d-ac6669e132a8","Type":"ContainerDied","Data":"2a9717e9f16ee89232fadcc80299291331ed5ad5d1c8a5263458de8973929d4a"} Oct 31 03:53:43.657891 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.657147 5490 generic.go:356] "Generic (PLEG): container finished" podID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerID="e80da2fd7fb9cb92560ec74ab6aa1a423731a1c2640319fd6b6fdb73d4762e62" exitCode=0 Oct 31 03:53:43.657891 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.657254 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" event={"ID":"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b","Type":"ContainerDied","Data":"e80da2fd7fb9cb92560ec74ab6aa1a423731a1c2640319fd6b6fdb73d4762e62"} Oct 31 03:53:43.657891 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.657314 5490 scope.go:122] "RemoveContainer" containerID="1d426b91989be27ab5889cf4a028ee30b26e35646ac0bb25bbc43a9986204d2c" Oct 31 03:53:43.666291 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.666221 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" event={"ID":"101e77ad-e8f5-4222-83ee-982ac3954dcc","Type":"ContainerDied","Data":"64382e4e17f65f1d673c6d45af5f19a58840ec3a10e32b2caf5262d45b8430c2"} Oct 31 03:53:43.666853 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.666823 5490 generic.go:356] "Generic (PLEG): container finished" podID="101e77ad-e8f5-4222-83ee-982ac3954dcc" containerID="64382e4e17f65f1d673c6d45af5f19a58840ec3a10e32b2caf5262d45b8430c2" exitCode=0 Oct 31 03:53:43.671498 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.671473 5490 generic.go:356] "Generic (PLEG): container finished" podID="5123bf1e-0ad6-4406-86b0-2b9a9e025a45" containerID="744a3ca1bfa7c5a9395b7d6b7b5d8b367ba6b48b2d8954fd459ae579b3c71f14" exitCode=0 Oct 31 03:53:43.671589 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.671545 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" event={"ID":"5123bf1e-0ad6-4406-86b0-2b9a9e025a45","Type":"ContainerDied","Data":"744a3ca1bfa7c5a9395b7d6b7b5d8b367ba6b48b2d8954fd459ae579b3c71f14"} Oct 31 03:53:43.681245 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.680876 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" Oct 31 03:53:43.853178 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.853149 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 03:53:43.856747 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:53:43.856670 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d426b91989be27ab5889cf4a028ee30b26e35646ac0bb25bbc43a9986204d2c\": container with ID starting with 1d426b91989be27ab5889cf4a028ee30b26e35646ac0bb25bbc43a9986204d2c not found: ID does not exist" containerID="1d426b91989be27ab5889cf4a028ee30b26e35646ac0bb25bbc43a9986204d2c" Oct 31 03:53:43.865926 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:43.865876 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:53:44.057318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.055524 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-token\" (UniqueName: \"kubernetes.io/projected/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-token\") pod \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " Oct 31 03:53:44.057318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.055592 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istiod-ca-cert\") pod \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " Oct 31 03:53:44.057318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.055698 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-credential-socket\") pod \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " Oct 31 03:53:44.057318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.055775 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-kube-api-access-prpt4\" (UniqueName: \"kubernetes.io/projected/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-kube-api-access-prpt4\") pod \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " Oct 31 03:53:44.057318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.055811 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-workload-socket\") pod \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " Oct 31 03:53:44.057318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.055891 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-podinfo\") pod \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " Oct 31 03:53:44.057318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.055948 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-envoy\") pod \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " Oct 31 03:53:44.057318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.056075 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-workload-certs\") pod \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " Oct 31 03:53:44.057318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.056183 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-data\") pod \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\" (UID: \"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b\") " Oct 31 03:53:44.063102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.061788 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-workload-certs" pod "a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" (UID: "a2dbe7ba-df3f-4b3f-a6c8-6e386172702b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:44.064200 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.063473 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-credential-socket" pod "a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" (UID: "a2dbe7ba-df3f-4b3f-a6c8-6e386172702b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:44.065089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.065051 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-workload-socket" pod "a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" (UID: "a2dbe7ba-df3f-4b3f-a6c8-6e386172702b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:44.065898 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.065851 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istiod-ca-cert" pod "a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" (UID: "a2dbe7ba-df3f-4b3f-a6c8-6e386172702b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:53:44.066505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.066476 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-data" pod "a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" (UID: "a2dbe7ba-df3f-4b3f-a6c8-6e386172702b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:44.079579 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.079527 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-kube-api-access-prpt4" pod "a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" (UID: "a2dbe7ba-df3f-4b3f-a6c8-6e386172702b"). InnerVolumeSpecName "kube-api-access-prpt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:44.083018 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.082949 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-envoy" pod "a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" (UID: "a2dbe7ba-df3f-4b3f-a6c8-6e386172702b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:44.083867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.083841 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:53:44.095879 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.095817 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-token" pod "a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" (UID: "a2dbe7ba-df3f-4b3f-a6c8-6e386172702b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:44.097512 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.097446 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-podinfo" pod "a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" (UID: "a2dbe7ba-df3f-4b3f-a6c8-6e386172702b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:53:44.176718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.168085 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.176718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.168113 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.176718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.168128 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.176718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.168142 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.176718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.168159 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prpt4\" (UniqueName: \"kubernetes.io/projected/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-kube-api-access-prpt4\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.176718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.168173 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.176718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.168186 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.176718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.168198 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.176718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.168213 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.268969 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.268932 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-credential-socket\") pod \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " Oct 31 03:53:44.269473 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.268984 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-workload-socket\") pod \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " Oct 31 03:53:44.269473 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.269025 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-data\") pod \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " Oct 31 03:53:44.269473 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.269135 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-workload-certs\") pod \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " Oct 31 03:53:44.269473 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.269331 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-kube-api-access-vfffj\" (UniqueName: \"kubernetes.io/projected/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-kube-api-access-vfffj\") pod \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " Oct 31 03:53:44.269473 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.269389 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-envoy\") pod \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " Oct 31 03:53:44.269473 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.269437 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-token\" (UniqueName: \"kubernetes.io/projected/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-token\") pod \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " Oct 31 03:53:44.272238 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.269524 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istiod-ca-cert\") pod \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " Oct 31 03:53:44.272238 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.269553 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-podinfo\") pod \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\" (UID: \"5123bf1e-0ad6-4406-86b0-2b9a9e025a45\") " Oct 31 03:53:44.334402 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.333749 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:53:44.371661 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.371631 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-workload-certs\") pod \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " Oct 31 03:53:44.371870 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.371693 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-envoy\") pod \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " Oct 31 03:53:44.371870 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.371722 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istiod-ca-cert\") pod \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " Oct 31 03:53:44.371870 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.371860 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-podinfo\") pod \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " Oct 31 03:53:44.372092 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.372054 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-credential-socket\") pod \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " Oct 31 03:53:44.372144 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.372135 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-workload-socket\") pod \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " Oct 31 03:53:44.372217 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.372204 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-kube-api-access-h5rrn\" (UniqueName: \"kubernetes.io/projected/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-kube-api-access-h5rrn\") pod \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " Oct 31 03:53:44.372265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.372252 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-data\") pod \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " Oct 31 03:53:44.372308 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.372284 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-token\" (UniqueName: \"kubernetes.io/projected/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-token\") pod \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\" (UID: \"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7\") " Oct 31 03:53:44.406782 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.406734 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-workload-certs" pod "d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" (UID: "d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:44.406947 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.406785 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:53:44.439224 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.439183 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-envoy" pod "d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" (UID: "d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:44.461092 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.461050 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-envoy" pod "5123bf1e-0ad6-4406-86b0-2b9a9e025a45" (UID: "5123bf1e-0ad6-4406-86b0-2b9a9e025a45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:44.473819 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.473791 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-credential-socket\") pod \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " Oct 31 03:53:44.473972 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.473859 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istiod-ca-cert\") pod \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " Oct 31 03:53:44.473972 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.473937 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-workload-certs\") pod \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " Oct 31 03:53:44.474087 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.473981 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-podinfo\") pod \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " Oct 31 03:53:44.474134 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.474125 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-data\") pod \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " Oct 31 03:53:44.474237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.474224 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-envoy\") pod \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " Oct 31 03:53:44.474305 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.474294 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-token\" (UniqueName: \"kubernetes.io/projected/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-token\") pod \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " Oct 31 03:53:44.474345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.474327 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcfb42f0-a328-4179-9a5d-ac6669e132a8-kube-api-access-fbgvj\" (UniqueName: \"kubernetes.io/projected/dcfb42f0-a328-4179-9a5d-ac6669e132a8-kube-api-access-fbgvj\") pod \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " Oct 31 03:53:44.534736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.474351 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-workload-socket\") pod \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\" (UID: \"dcfb42f0-a328-4179-9a5d-ac6669e132a8\") " Oct 31 03:53:44.534736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.475008 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.534736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.475024 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.534736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.475033 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.535091 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.535052 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istiod-ca-cert" pod "d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" (UID: "d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:53:44.535481 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.535452 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-workload-socket" pod "d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" (UID: "d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:44.577592 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.577552 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.577592 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.577587 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.580638 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.580549 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-data" pod "d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" (UID: "d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:44.580798 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.580659 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-credential-socket" pod "d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" (UID: "d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:44.664951 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.664906 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-podinfo" pod "d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" (UID: "d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:53:44.679884 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.679850 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.679884 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.679880 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.679884 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.679893 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.687508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.687465 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" event={"ID":"fd89b915-fe3a-4a8a-aaf5-6927987b2e0d","Type":"ContainerStarted","Data":"2989195b546436a4afe64dc10f9e65b9d657df9f12126456d023e319797bf05b"} Oct 31 03:53:44.687694 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.687589 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:53:44.689506 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.689481 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" event={"ID":"ce529bc9-a44d-4808-8d2c-996e14f88a54","Type":"ContainerStarted","Data":"b0effe8cb93185235388a1bfd913de6c6ef3fcd7c7129ef68a20b31415489d30"} Oct 31 03:53:44.689626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.689600 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" Oct 31 03:53:44.690909 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.690881 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" Oct 31 03:53:44.690909 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.690892 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p" event={"ID":"5123bf1e-0ad6-4406-86b0-2b9a9e025a45","Type":"ContainerDied","Data":"8ab255d063fd92dd1de2fca11a413ce665318e934de0be896332fa49b1ff727e"} Oct 31 03:53:44.691043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.690942 5490 scope.go:122] "RemoveContainer" containerID="744a3ca1bfa7c5a9395b7d6b7b5d8b367ba6b48b2d8954fd459ae579b3c71f14" Oct 31 03:53:44.692598 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.692574 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" event={"ID":"824618ba-ce21-4eeb-bf27-f1a861327863","Type":"ContainerStarted","Data":"41f67ea9dd6fb51c2bb5ab18a755f9a86020168a384eca37e881b3afe0bceb47"} Oct 31 03:53:44.695095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.695073 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zngp8" event={"ID":"08da34c7-c69a-4eb9-848a-399cab14840e","Type":"ContainerStarted","Data":"9bce376154617c1607b8cbb3a2e13a2b17733dd9b4ccf80764348e977eb85403"} Oct 31 03:53:44.696753 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.696731 5490 generic.go:356] "Generic (PLEG): container finished" podID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerID="8212151b6af29b681d3346cd5171621f62899608c3f910e235e612a058935331" exitCode=0 Oct 31 03:53:44.696822 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.696798 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" event={"ID":"ef5f9523-ec4a-4372-a327-009c9fc40563","Type":"ContainerDied","Data":"8212151b6af29b681d3346cd5171621f62899608c3f910e235e612a058935331"} Oct 31 03:53:44.696859 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.696836 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" event={"ID":"ef5f9523-ec4a-4372-a327-009c9fc40563","Type":"ContainerDied","Data":"c0a44c734d1c7a3e72e60683e7fe876ee329adc5db003ea6fc0273ab45413bd6"} Oct 31 03:53:44.696917 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.696858 5490 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0a44c734d1c7a3e72e60683e7fe876ee329adc5db003ea6fc0273ab45413bd6" Oct 31 03:53:44.698743 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.698705 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586ddd746f-nv8gm" event={"ID":"2238dc08-3a3e-426e-a8f7-22bec5f016c8","Type":"ContainerStarted","Data":"8d2b0c5a39c5ebe143891d27760c13a3b12577916375a95cdb0f7e959edac463"} Oct 31 03:53:44.698852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.698838 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-586ddd746f-nv8gm" Oct 31 03:53:44.700207 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.700132 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-token" pod "d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" (UID: "d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:44.700207 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.700143 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-kube-api-access-h5rrn" pod "d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" (UID: "d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7"). InnerVolumeSpecName "kube-api-access-h5rrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:44.700386 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.700273 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:53:44.700471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.700446 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" event={"ID":"05bbcb81-55ce-4387-89c3-f6350fe8636f","Type":"ContainerStarted","Data":"ce4f1160f273d64b012eb62dba9a79a15defc4613a3b6e8efbfe985e6ec2d6d0"} Oct 31 03:53:44.702551 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.702532 5490 generic.go:356] "Generic (PLEG): container finished" podID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" containerID="042b2b5699dc8a0e8f1b9b4fc6fa574f84cbd93097f524eac8d7af8f87354bb6" exitCode=0 Oct 31 03:53:44.702649 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.702597 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" event={"ID":"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666","Type":"ContainerDied","Data":"042b2b5699dc8a0e8f1b9b4fc6fa574f84cbd93097f524eac8d7af8f87354bb6"} Oct 31 03:53:44.702649 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.702634 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" event={"ID":"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666","Type":"ContainerDied","Data":"fafe8613676fc4583983dc2607627e434123b8a96773fcc8a6ff3c0275182239"} Oct 31 03:53:44.702761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.702653 5490 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fafe8613676fc4583983dc2607627e434123b8a96773fcc8a6ff3c0275182239" Oct 31 03:53:44.703990 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.703971 5490 generic.go:356] "Generic (PLEG): container finished" podID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerID="e24242cc752af656810ca6373322bafdec557f3a573129be7802d1166a376163" exitCode=0 Oct 31 03:53:44.704087 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.704003 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" Oct 31 03:53:44.704087 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.704079 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" event={"ID":"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7","Type":"ContainerDied","Data":"e24242cc752af656810ca6373322bafdec557f3a573129be7802d1166a376163"} Oct 31 03:53:44.704200 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.704119 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv" event={"ID":"d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7","Type":"ContainerDied","Data":"4bafe6780a261405095d6783541ec7c61c313ecf175969ed1c57576bca444527"} Oct 31 03:53:44.705551 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.705533 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" event={"ID":"dcfb42f0-a328-4179-9a5d-ac6669e132a8","Type":"ContainerDied","Data":"0edb14b34644c09a53b2df5a24c57640503edec91fdb6ea372ab1b7a8194d606"} Oct 31 03:53:44.705619 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.705594 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h" Oct 31 03:53:44.706978 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.706961 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" event={"ID":"a2dbe7ba-df3f-4b3f-a6c8-6e386172702b","Type":"ContainerDied","Data":"664c32076431a9e2b8e694813314df0fb14e1d9f435827bf764ac2dccabed58b"} Oct 31 03:53:44.707039 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.707029 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" Oct 31 03:53:44.711914 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.711885 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" event={"ID":"101e77ad-e8f5-4222-83ee-982ac3954dcc","Type":"ContainerDied","Data":"8d675da5765af158d1b595feb0498851bfdff08afdd7d210809b009bb8363786"} Oct 31 03:53:44.711994 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.711927 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8" Oct 31 03:53:44.747746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.740327 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-workload-certs" pod "5123bf1e-0ad6-4406-86b0-2b9a9e025a45" (UID: "5123bf1e-0ad6-4406-86b0-2b9a9e025a45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:44.747746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.740402 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-workload-socket" pod "5123bf1e-0ad6-4406-86b0-2b9a9e025a45" (UID: "5123bf1e-0ad6-4406-86b0-2b9a9e025a45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:44.828896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.778288 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-data" pod "5123bf1e-0ad6-4406-86b0-2b9a9e025a45" (UID: "5123bf1e-0ad6-4406-86b0-2b9a9e025a45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:44.828896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.781295 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-podinfo\") pod \"101e77ad-e8f5-4222-83ee-982ac3954dcc\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " Oct 31 03:53:44.828896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.781558 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-workload-socket\") pod \"101e77ad-e8f5-4222-83ee-982ac3954dcc\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " Oct 31 03:53:44.828896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.781753 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-workload-certs\") pod \"101e77ad-e8f5-4222-83ee-982ac3954dcc\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " Oct 31 03:53:44.828896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.781812 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-credential-socket\") pod \"101e77ad-e8f5-4222-83ee-982ac3954dcc\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " Oct 31 03:53:44.828896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.781874 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-token\" (UniqueName: \"kubernetes.io/projected/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-token\") pod \"101e77ad-e8f5-4222-83ee-982ac3954dcc\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " Oct 31 03:53:44.828896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.781941 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-envoy\") pod \"101e77ad-e8f5-4222-83ee-982ac3954dcc\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " Oct 31 03:53:44.828896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.782060 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101e77ad-e8f5-4222-83ee-982ac3954dcc-kube-api-access-b7z8h\" (UniqueName: \"kubernetes.io/projected/101e77ad-e8f5-4222-83ee-982ac3954dcc-kube-api-access-b7z8h\") pod \"101e77ad-e8f5-4222-83ee-982ac3954dcc\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " Oct 31 03:53:44.828896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.782126 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-data\") pod \"101e77ad-e8f5-4222-83ee-982ac3954dcc\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " Oct 31 03:53:44.828896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.782202 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101e77ad-e8f5-4222-83ee-982ac3954dcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/101e77ad-e8f5-4222-83ee-982ac3954dcc-istiod-ca-cert\") pod \"101e77ad-e8f5-4222-83ee-982ac3954dcc\" (UID: \"101e77ad-e8f5-4222-83ee-982ac3954dcc\") " Oct 31 03:53:44.828896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.783348 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.828896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.783376 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.828896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.783393 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5rrn\" (UniqueName: \"kubernetes.io/projected/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-kube-api-access-h5rrn\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.828896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.783403 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.828896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.783415 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:44.828896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.792984 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 03:53:44.828896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.793031 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 03:53:44.980714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.945654 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-token" pod "5123bf1e-0ad6-4406-86b0-2b9a9e025a45" (UID: "5123bf1e-0ad6-4406-86b0-2b9a9e025a45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:44.980714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.945730 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-kube-api-access-vfffj" pod "5123bf1e-0ad6-4406-86b0-2b9a9e025a45" (UID: "5123bf1e-0ad6-4406-86b0-2b9a9e025a45"). InnerVolumeSpecName "kube-api-access-vfffj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:44.980714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.945773 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-data" pod "dcfb42f0-a328-4179-9a5d-ac6669e132a8" (UID: "dcfb42f0-a328-4179-9a5d-ac6669e132a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.007708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.987315 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.007708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.987342 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfffj\" (UniqueName: \"kubernetes.io/projected/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-kube-api-access-vfffj\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.007708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:44.987355 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.007708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.001351 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istiod-ca-cert" pod "5123bf1e-0ad6-4406-86b0-2b9a9e025a45" (UID: "5123bf1e-0ad6-4406-86b0-2b9a9e025a45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:53:45.044748 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.026784 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-data" pod "101e77ad-e8f5-4222-83ee-982ac3954dcc" (UID: "101e77ad-e8f5-4222-83ee-982ac3954dcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.044748 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.026811 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-workload-socket" pod "101e77ad-e8f5-4222-83ee-982ac3954dcc" (UID: "101e77ad-e8f5-4222-83ee-982ac3954dcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.044748 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.026798 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-credential-socket" pod "5123bf1e-0ad6-4406-86b0-2b9a9e025a45" (UID: "5123bf1e-0ad6-4406-86b0-2b9a9e025a45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.044748 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.026894 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-podinfo" pod "5123bf1e-0ad6-4406-86b0-2b9a9e025a45" (UID: "5123bf1e-0ad6-4406-86b0-2b9a9e025a45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:53:45.044748 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.027088 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101e77ad-e8f5-4222-83ee-982ac3954dcc-istiod-ca-cert" pod "101e77ad-e8f5-4222-83ee-982ac3954dcc" (UID: "101e77ad-e8f5-4222-83ee-982ac3954dcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:53:45.044748 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.027422 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-workload-socket" pod "dcfb42f0-a328-4179-9a5d-ac6669e132a8" (UID: "dcfb42f0-a328-4179-9a5d-ac6669e132a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.093055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.089648 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.093055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.089690 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.093055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.089705 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.093055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.089718 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/101e77ad-e8f5-4222-83ee-982ac3954dcc-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.093055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.089730 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.093055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.089740 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.093055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.089748 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5123bf1e-0ad6-4406-86b0-2b9a9e025a45-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.093055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.090960 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-credential-socket" pod "dcfb42f0-a328-4179-9a5d-ac6669e132a8" (UID: "dcfb42f0-a328-4179-9a5d-ac6669e132a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.093055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.091008 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-envoy" pod "101e77ad-e8f5-4222-83ee-982ac3954dcc" (UID: "101e77ad-e8f5-4222-83ee-982ac3954dcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.093055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.091048 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-workload-certs" pod "dcfb42f0-a328-4179-9a5d-ac6669e132a8" (UID: "dcfb42f0-a328-4179-9a5d-ac6669e132a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.093055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.091065 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-credential-socket" pod "101e77ad-e8f5-4222-83ee-982ac3954dcc" (UID: "101e77ad-e8f5-4222-83ee-982ac3954dcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.093055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.091131 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-token" pod "101e77ad-e8f5-4222-83ee-982ac3954dcc" (UID: "101e77ad-e8f5-4222-83ee-982ac3954dcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:45.093055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.091172 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101e77ad-e8f5-4222-83ee-982ac3954dcc-kube-api-access-b7z8h" pod "101e77ad-e8f5-4222-83ee-982ac3954dcc" (UID: "101e77ad-e8f5-4222-83ee-982ac3954dcc"). InnerVolumeSpecName "kube-api-access-b7z8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:45.142610 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.141645 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-workload-certs" pod "101e77ad-e8f5-4222-83ee-982ac3954dcc" (UID: "101e77ad-e8f5-4222-83ee-982ac3954dcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.205997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.191667 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.205997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.191747 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.205997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.191761 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.205997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.191775 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.205997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.191790 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.205997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.191804 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7z8h\" (UniqueName: \"kubernetes.io/projected/101e77ad-e8f5-4222-83ee-982ac3954dcc-kube-api-access-b7z8h\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.205997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.191818 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.205997 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.205501 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:45.253177 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.206746 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-token" pod "dcfb42f0-a328-4179-9a5d-ac6669e132a8" (UID: "dcfb42f0-a328-4179-9a5d-ac6669e132a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:45.255670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.255219 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-podinfo" pod "101e77ad-e8f5-4222-83ee-982ac3954dcc" (UID: "101e77ad-e8f5-4222-83ee-982ac3954dcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:53:45.255670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.255219 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcfb42f0-a328-4179-9a5d-ac6669e132a8-kube-api-access-fbgvj" pod "dcfb42f0-a328-4179-9a5d-ac6669e132a8" (UID: "dcfb42f0-a328-4179-9a5d-ac6669e132a8"). InnerVolumeSpecName "kube-api-access-fbgvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:45.255670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.255309 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istiod-ca-cert" pod "dcfb42f0-a328-4179-9a5d-ac6669e132a8" (UID: "dcfb42f0-a328-4179-9a5d-ac6669e132a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:53:45.255670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.255343 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:53:45.255670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.255331 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-podinfo" pod "dcfb42f0-a328-4179-9a5d-ac6669e132a8" (UID: "dcfb42f0-a328-4179-9a5d-ac6669e132a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:53:45.255670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.255410 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-envoy" pod "dcfb42f0-a328-4179-9a5d-ac6669e132a8" (UID: "dcfb42f0-a328-4179-9a5d-ac6669e132a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.287019 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.286954 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:53:45.287137 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.287116 5490 scope.go:122] "RemoveContainer" containerID="6416be98e88ea871330e8fb38a3a650458ed0aebc40ff174c0a8920654a91bc5" Oct 31 03:53:45.292342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.292312 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-data\") pod \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " Oct 31 03:53:45.292508 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.292473 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-envoy\") pod \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " Oct 31 03:53:45.292586 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.292521 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-workload-socket\") pod \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " Oct 31 03:53:45.292644 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.292584 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-kube-api-access-6gsbc\" (UniqueName: \"kubernetes.io/projected/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-kube-api-access-6gsbc\") pod \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " Oct 31 03:53:45.292743 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.292672 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-token\" (UniqueName: \"kubernetes.io/projected/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-token\") pod \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " Oct 31 03:53:45.292815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.292752 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istiod-ca-cert\") pod \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " Oct 31 03:53:45.292870 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.292835 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-podinfo\") pod \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " Oct 31 03:53:45.292954 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.292939 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-workload-certs\") pod \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " Oct 31 03:53:45.293010 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.292971 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-credential-socket\") pod \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\" (UID: \"6e563544-8d00-4f0f-bd8a-c0cfc2c5c666\") " Oct 31 03:53:45.293439 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.293422 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.293439 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.293439 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.293564 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.293449 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbgvj\" (UniqueName: \"kubernetes.io/projected/dcfb42f0-a328-4179-9a5d-ac6669e132a8-kube-api-access-fbgvj\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.293564 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.293458 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.293564 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.293467 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcfb42f0-a328-4179-9a5d-ac6669e132a8-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.293564 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.293475 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/101e77ad-e8f5-4222-83ee-982ac3954dcc-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.318672 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.297101 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:53:45.318672 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.298975 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 31 03:53:45.318672 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.299022 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" podUID="824618ba-ce21-4eeb-bf27-f1a861327863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 31 03:53:45.318672 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.302964 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv"] Oct 31 03:53:45.318672 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.308692 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-6gwcv"] Oct 31 03:53:45.351784 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.351613 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-data" pod "6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" (UID: "6e563544-8d00-4f0f-bd8a-c0cfc2c5c666"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.355186 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.355129 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" podStartSLOduration=173.355110027 podStartE2EDuration="2m53.355110027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:45.349803076 +0000 UTC m=+2259.884498028" watchObservedRunningTime="2026-10-31 03:53:45.355110027 +0000 UTC m=+2259.889804985" Oct 31 03:53:45.395415 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.395382 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-data\") pod \"ef5f9523-ec4a-4372-a327-009c9fc40563\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " Oct 31 03:53:45.395593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.395438 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-workload-certs\") pod \"ef5f9523-ec4a-4372-a327-009c9fc40563\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " Oct 31 03:53:45.395593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.395500 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-credential-socket\") pod \"ef5f9523-ec4a-4372-a327-009c9fc40563\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " Oct 31 03:53:45.395810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.395700 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-token\" (UniqueName: \"kubernetes.io/projected/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-token\") pod \"ef5f9523-ec4a-4372-a327-009c9fc40563\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " Oct 31 03:53:45.395810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.395798 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-workload-socket\") pod \"ef5f9523-ec4a-4372-a327-009c9fc40563\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " Oct 31 03:53:45.395924 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.395823 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-envoy\") pod \"ef5f9523-ec4a-4372-a327-009c9fc40563\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " Oct 31 03:53:45.395924 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.395887 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-podinfo\") pod \"ef5f9523-ec4a-4372-a327-009c9fc40563\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " Oct 31 03:53:45.396053 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.395926 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef5f9523-ec4a-4372-a327-009c9fc40563-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef5f9523-ec4a-4372-a327-009c9fc40563-istiod-ca-cert\") pod \"ef5f9523-ec4a-4372-a327-009c9fc40563\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " Oct 31 03:53:45.396053 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.395982 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef5f9523-ec4a-4372-a327-009c9fc40563-kube-api-access-vmmfb\" (UniqueName: \"kubernetes.io/projected/ef5f9523-ec4a-4372-a327-009c9fc40563-kube-api-access-vmmfb\") pod \"ef5f9523-ec4a-4372-a327-009c9fc40563\" (UID: \"ef5f9523-ec4a-4372-a327-009c9fc40563\") " Oct 31 03:53:45.396710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.396656 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.398462 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.398417 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" podStartSLOduration=158.39840375 podStartE2EDuration="2m38.39840375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:45.394994091 +0000 UTC m=+2259.929689047" watchObservedRunningTime="2026-10-31 03:53:45.39840375 +0000 UTC m=+2259.933098767" Oct 31 03:53:45.413061 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.412924 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-workload-certs" pod "ef5f9523-ec4a-4372-a327-009c9fc40563" (UID: "ef5f9523-ec4a-4372-a327-009c9fc40563"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.413061 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.412938 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-credential-socket" pod "ef5f9523-ec4a-4372-a327-009c9fc40563" (UID: "ef5f9523-ec4a-4372-a327-009c9fc40563"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.413506 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.413481 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-workload-socket" pod "ef5f9523-ec4a-4372-a327-009c9fc40563" (UID: "ef5f9523-ec4a-4372-a327-009c9fc40563"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.413656 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.413641 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-data" pod "ef5f9523-ec4a-4372-a327-009c9fc40563" (UID: "ef5f9523-ec4a-4372-a327-009c9fc40563"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.429568 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.428817 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:53:45.436289 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.436252 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 31 03:53:45.436464 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.436322 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" podUID="05bbcb81-55ce-4387-89c3-f6350fe8636f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 31 03:53:45.461811 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.461764 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-podinfo" pod "ef5f9523-ec4a-4372-a327-009c9fc40563" (UID: "ef5f9523-ec4a-4372-a327-009c9fc40563"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:53:45.469898 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.469717 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-envoy" pod "ef5f9523-ec4a-4372-a327-009c9fc40563" (UID: "ef5f9523-ec4a-4372-a327-009c9fc40563"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.469898 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.469857 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-token" pod "ef5f9523-ec4a-4372-a327-009c9fc40563" (UID: "ef5f9523-ec4a-4372-a327-009c9fc40563"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:45.489776 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.489735 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh"] Oct 31 03:53:45.537941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.496851 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-hmwnh"] Oct 31 03:53:45.537941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.498526 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.537941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.498552 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.537941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.498571 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.537941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.498584 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.537941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.498597 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.537941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.498612 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef5f9523-ec4a-4372-a327-009c9fc40563-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.537941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.498624 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef5f9523-ec4a-4372-a327-009c9fc40563-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.537941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.525741 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p"] Oct 31 03:53:45.537941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.534458 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gfr8p"] Oct 31 03:53:45.547131 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.547072 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h"] Oct 31 03:53:45.551804 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.551772 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-t6x8h"] Oct 31 03:53:45.561278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.561249 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8"] Oct 31 03:53:45.566013 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.565986 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-cx4p8"] Oct 31 03:53:45.578461 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.578418 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-kube-api-access-6gsbc" pod "6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" (UID: "6e563544-8d00-4f0f-bd8a-c0cfc2c5c666"). InnerVolumeSpecName "kube-api-access-6gsbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:45.599843 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.599810 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gsbc\" (UniqueName: \"kubernetes.io/projected/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-kube-api-access-6gsbc\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.606547 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.606443 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-podinfo" pod "6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" (UID: "6e563544-8d00-4f0f-bd8a-c0cfc2c5c666"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:53:45.607224 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.607176 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-token" pod "6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" (UID: "6e563544-8d00-4f0f-bd8a-c0cfc2c5c666"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:45.608212 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.607707 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-workload-socket" pod "6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" (UID: "6e563544-8d00-4f0f-bd8a-c0cfc2c5c666"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.608212 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.607749 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="101e77ad-e8f5-4222-83ee-982ac3954dcc" path="/var/lib/kubelet/pods/101e77ad-e8f5-4222-83ee-982ac3954dcc/volumes" Oct 31 03:53:45.612268 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.612076 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-credential-socket" pod "6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" (UID: "6e563544-8d00-4f0f-bd8a-c0cfc2c5c666"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.612268 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.612115 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-workload-certs" pod "6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" (UID: "6e563544-8d00-4f0f-bd8a-c0cfc2c5c666"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.615705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.612691 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istiod-ca-cert" pod "6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" (UID: "6e563544-8d00-4f0f-bd8a-c0cfc2c5c666"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:53:45.615705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.612818 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5f9523-ec4a-4372-a327-009c9fc40563-kube-api-access-vmmfb" pod "ef5f9523-ec4a-4372-a327-009c9fc40563" (UID: "ef5f9523-ec4a-4372-a327-009c9fc40563"). InnerVolumeSpecName "kube-api-access-vmmfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:45.615705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.613390 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5123bf1e-0ad6-4406-86b0-2b9a9e025a45" path="/var/lib/kubelet/pods/5123bf1e-0ad6-4406-86b0-2b9a9e025a45/volumes" Oct 31 03:53:45.615705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.614062 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5f9523-ec4a-4372-a327-009c9fc40563-istiod-ca-cert" pod "ef5f9523-ec4a-4372-a327-009c9fc40563" (UID: "ef5f9523-ec4a-4372-a327-009c9fc40563"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:53:45.615705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.615025 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" path="/var/lib/kubelet/pods/a2dbe7ba-df3f-4b3f-a6c8-6e386172702b/volumes" Oct 31 03:53:45.616309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.616277 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" path="/var/lib/kubelet/pods/d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7/volumes" Oct 31 03:53:45.618090 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.617023 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcfb42f0-a328-4179-9a5d-ac6669e132a8" path="/var/lib/kubelet/pods/dcfb42f0-a328-4179-9a5d-ac6669e132a8/volumes" Oct 31 03:53:45.666908 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.666769 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-envoy" pod "6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" (UID: "6e563544-8d00-4f0f-bd8a-c0cfc2c5c666"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:45.667794 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.667738 5490 scope.go:122] "RemoveContainer" containerID="5095b4b424c27e0d1b4508340a2ef081f59d900607606f5547cd9d9696dd72b7" Oct 31 03:53:45.723938 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.715846 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.723938 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.715887 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.723938 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.715904 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.723938 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.715919 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.723938 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.715941 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.723938 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.715960 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef5f9523-ec4a-4372-a327-009c9fc40563-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.723938 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.716001 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.723938 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.716017 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.723938 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.716038 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmmfb\" (UniqueName: \"kubernetes.io/projected/ef5f9523-ec4a-4372-a327-009c9fc40563-kube-api-access-vmmfb\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:45.738079 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.738048 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt" Oct 31 03:53:45.740335 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.740306 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw" Oct 31 03:53:45.769334 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.769307 5490 scope.go:122] "RemoveContainer" containerID="e24242cc752af656810ca6373322bafdec557f3a573129be7802d1166a376163" Oct 31 03:53:45.785526 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.785489 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw"] Oct 31 03:53:45.790727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.790573 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-k9bcw"] Oct 31 03:53:45.796575 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.796546 5490 scope.go:122] "RemoveContainer" containerID="e24242cc752af656810ca6373322bafdec557f3a573129be7802d1166a376163" Oct 31 03:53:45.797088 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:53:45.797061 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24242cc752af656810ca6373322bafdec557f3a573129be7802d1166a376163\": container with ID starting with e24242cc752af656810ca6373322bafdec557f3a573129be7802d1166a376163 not found: ID does not exist" containerID="e24242cc752af656810ca6373322bafdec557f3a573129be7802d1166a376163" Oct 31 03:53:45.797318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.797292 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24242cc752af656810ca6373322bafdec557f3a573129be7802d1166a376163"} err="failed to get container status \"e24242cc752af656810ca6373322bafdec557f3a573129be7802d1166a376163\": rpc error: code = NotFound desc = could not find container \"e24242cc752af656810ca6373322bafdec557f3a573129be7802d1166a376163\": container with ID starting with e24242cc752af656810ca6373322bafdec557f3a573129be7802d1166a376163 not found: ID does not exist" Oct 31 03:53:45.797431 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.797420 5490 scope.go:122] "RemoveContainer" containerID="2a9717e9f16ee89232fadcc80299291331ed5ad5d1c8a5263458de8973929d4a" Oct 31 03:53:45.801802 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.801782 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt"] Oct 31 03:53:45.813781 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.813737 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-n2kpt"] Oct 31 03:53:45.817380 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.817327 5490 scope.go:122] "RemoveContainer" containerID="e80da2fd7fb9cb92560ec74ab6aa1a423731a1c2640319fd6b6fdb73d4762e62" Oct 31 03:53:45.897509 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:45.897097 5490 scope.go:122] "RemoveContainer" containerID="64382e4e17f65f1d673c6d45af5f19a58840ec3a10e32b2caf5262d45b8430c2" Oct 31 03:53:46.297249 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:46.297217 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 31 03:53:46.297702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:46.297274 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" podUID="824618ba-ce21-4eeb-bf27-f1a861327863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 31 03:53:46.425308 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:46.425275 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 31 03:53:46.425449 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:46.425330 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" podUID="05bbcb81-55ce-4387-89c3-f6350fe8636f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 31 03:53:46.784741 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:46.784453 5490 generic.go:356] "Generic (PLEG): container finished" podID="55f29950-78b4-4a52-818d-dc235e4631fc" containerID="20c1a8fe58f1b857d42ccc0528bd0a4c43f1aa5bbcec5f42f3409e3ac599573f" exitCode=137 Oct 31 03:53:46.784741 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:46.784567 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" event={"ID":"55f29950-78b4-4a52-818d-dc235e4631fc","Type":"ContainerDied","Data":"20c1a8fe58f1b857d42ccc0528bd0a4c43f1aa5bbcec5f42f3409e3ac599573f"} Oct 31 03:53:46.790542 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:46.790520 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ttmkb_400b676a-a3a0-459d-9da6-8d30b2d3b497/csi-driver/0.log" Oct 31 03:53:46.790932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:46.790898 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" event={"ID":"400b676a-a3a0-459d-9da6-8d30b2d3b497","Type":"ContainerStarted","Data":"2ab2d6a24ba484699827eca8b9cc87ccf071e41991f0af853f3063faebaf083c"} Oct 31 03:53:46.791024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:46.790982 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttmkb" event={"ID":"400b676a-a3a0-459d-9da6-8d30b2d3b497","Type":"ContainerStarted","Data":"c4eb2aeef2cf906a2b86351371917a8a2e765b9f684e787a8a96e7c5cc6c1a5c"} Oct 31 03:53:46.794210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:46.794186 5490 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 03:53:47.210062 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:53:47.210013 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:53:47.297408 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:47.297371 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 31 03:53:47.297812 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:47.297425 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" podUID="824618ba-ce21-4eeb-bf27-f1a861327863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 31 03:53:47.425640 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:47.425602 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 31 03:53:47.425846 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:47.425654 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" podUID="05bbcb81-55ce-4387-89c3-f6350fe8636f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 31 03:53:47.593336 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:47.593299 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" path="/var/lib/kubelet/pods/6e563544-8d00-4f0f-bd8a-c0cfc2c5c666/volumes" Oct 31 03:53:47.595217 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:47.595190 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef5f9523-ec4a-4372-a327-009c9fc40563" path="/var/lib/kubelet/pods/ef5f9523-ec4a-4372-a327-009c9fc40563/volumes" Oct 31 03:53:47.781761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:47.779762 5490 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:53:46.794201869Z","UUID":"2eb65596-1824-489d-b9f1-ed1d509764d4","Handler":null,"Name":"","Endpoint":""} Oct 31 03:53:47.781761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:47.781294 5490 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 03:53:47.781761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:47.781332 5490 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 03:53:47.808007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:47.807967 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" event={"ID":"55f29950-78b4-4a52-818d-dc235e4631fc","Type":"ContainerStarted","Data":"578d31fb3be5a1612a1699d364d98f4fade49fa7904536aa6b7d9fa8b32f3c48"} Oct 31 03:53:47.808182 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:47.808053 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:53:48.297361 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:48.297325 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 31 03:53:48.297599 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:48.297576 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" podUID="824618ba-ce21-4eeb-bf27-f1a861327863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 31 03:53:48.425442 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:48.425407 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 31 03:53:48.425597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:48.425463 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" podUID="05bbcb81-55ce-4387-89c3-f6350fe8636f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 31 03:53:48.680423 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:48.680387 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-bdjls" Oct 31 03:53:48.684994 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:48.684971 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-b949d9bf6-8b6xc" Oct 31 03:53:49.297397 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:49.297361 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 31 03:53:49.297568 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:49.297418 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" podUID="824618ba-ce21-4eeb-bf27-f1a861327863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 31 03:53:49.396927 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:49.396887 5490 scope.go:122] "RemoveContainer" containerID="8212151b6af29b681d3346cd5171621f62899608c3f910e235e612a058935331" Oct 31 03:53:49.412257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:49.412199 5490 scope.go:122] "RemoveContainer" containerID="042b2b5699dc8a0e8f1b9b4fc6fa574f84cbd93097f524eac8d7af8f87354bb6" Oct 31 03:53:49.425211 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:49.425183 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 31 03:53:49.425329 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:49.425237 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" podUID="05bbcb81-55ce-4387-89c3-f6350fe8636f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 31 03:53:50.297179 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:50.297139 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:53:50.297365 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:50.297337 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 31 03:53:50.297408 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:50.297385 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" podUID="824618ba-ce21-4eeb-bf27-f1a861327863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 31 03:53:50.425323 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:50.425285 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 31 03:53:50.425749 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:50.425342 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" podUID="05bbcb81-55ce-4387-89c3-f6350fe8636f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 31 03:53:51.093442 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:51.093221 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:51.220282 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:51.220182 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:51.297485 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:51.297451 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body= Oct 31 03:53:51.297813 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:51.297765 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" podUID="824618ba-ce21-4eeb-bf27-f1a861327863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" Oct 31 03:53:51.428558 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:51.428521 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 31 03:53:51.429196 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:51.429141 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" podUID="05bbcb81-55ce-4387-89c3-f6350fe8636f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 31 03:53:51.576460 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:51.576417 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:52.302358 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:52.302329 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:53:52.303384 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:52.303366 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:53:52.430803 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:52.430770 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:53:52.431273 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:52.431258 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:53:52.432037 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:52.432019 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:53:52.962989 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:52.962950 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7"] Oct 31 03:53:52.963224 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:52.963164 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" podUID="cdf49e86-d2b1-4888-8475-8a85dd69a6f3" containerName="istio-proxy" containerID="cri-o://40c306e87ba45cc24f8f8a23e301f90588e657aafd2470a31afc320248c53bca" gracePeriod=30 Oct 31 03:53:53.681157 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:53.681119 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-f2tk8" Oct 31 03:53:54.042301 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:54.042213 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:54.299641 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:54.299540 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:55.742770 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:55.742729 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-vtdtl" Oct 31 03:53:55.743308 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:55.742810 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-586ddd746f-nv8gm" Oct 31 03:53:57.669158 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:57.669113 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:53:57.951203 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:57.951122 5490 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:53:57.951203 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:57.951182 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" podUID="cdf49e86-d2b1-4888-8475-8a85dd69a6f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:53:58.253493 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.253467 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:53:58.393017 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.392981 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-workload-socket\") pod \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " Oct 31 03:53:58.393228 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.393052 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-kube-api-access-rrmhl\" (UniqueName: \"kubernetes.io/projected/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-kube-api-access-rrmhl\") pod \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " Oct 31 03:53:58.393838 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.393797 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-credential-socket\") pod \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " Oct 31 03:53:58.394400 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.394369 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istiod-ca-cert\") pod \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " Oct 31 03:53:58.395057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.394482 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-data\") pod \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " Oct 31 03:53:58.395057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.394792 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-podinfo\") pod \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " Oct 31 03:53:58.395057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.394895 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-workload-certs\") pod \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " Oct 31 03:53:58.395057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.394942 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-envoy\") pod \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " Oct 31 03:53:58.395057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.395001 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-token\" (UniqueName: \"kubernetes.io/projected/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-token\") pod \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\" (UID: \"cdf49e86-d2b1-4888-8475-8a85dd69a6f3\") " Oct 31 03:53:58.396240 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.395965 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-credential-socket" pod "cdf49e86-d2b1-4888-8475-8a85dd69a6f3" (UID: "cdf49e86-d2b1-4888-8475-8a85dd69a6f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:58.396240 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.396112 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-data" pod "cdf49e86-d2b1-4888-8475-8a85dd69a6f3" (UID: "cdf49e86-d2b1-4888-8475-8a85dd69a6f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:58.396240 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.396012 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-workload-socket" pod "cdf49e86-d2b1-4888-8475-8a85dd69a6f3" (UID: "cdf49e86-d2b1-4888-8475-8a85dd69a6f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:58.396825 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.396791 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-workload-certs" pod "cdf49e86-d2b1-4888-8475-8a85dd69a6f3" (UID: "cdf49e86-d2b1-4888-8475-8a85dd69a6f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:58.397043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.397020 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istiod-ca-cert" pod "cdf49e86-d2b1-4888-8475-8a85dd69a6f3" (UID: "cdf49e86-d2b1-4888-8475-8a85dd69a6f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:53:58.397920 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.397897 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-kube-api-access-rrmhl" pod "cdf49e86-d2b1-4888-8475-8a85dd69a6f3" (UID: "cdf49e86-d2b1-4888-8475-8a85dd69a6f3"). InnerVolumeSpecName "kube-api-access-rrmhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:58.398895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.398868 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-podinfo" pod "cdf49e86-d2b1-4888-8475-8a85dd69a6f3" (UID: "cdf49e86-d2b1-4888-8475-8a85dd69a6f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:53:58.399113 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.399092 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-token" pod "cdf49e86-d2b1-4888-8475-8a85dd69a6f3" (UID: "cdf49e86-d2b1-4888-8475-8a85dd69a6f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:53:58.399385 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.399247 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-envoy" pod "cdf49e86-d2b1-4888-8475-8a85dd69a6f3" (UID: "cdf49e86-d2b1-4888-8475-8a85dd69a6f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:53:58.497597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.497503 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:58.497597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.497543 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:58.497597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.497556 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:58.497597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.497565 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:58.497597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.497575 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:58.497597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.497584 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:58.497597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.497592 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:58.497597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.497601 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:58.498150 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.497612 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrmhl\" (UniqueName: \"kubernetes.io/projected/cdf49e86-d2b1-4888-8475-8a85dd69a6f3-kube-api-access-rrmhl\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:53:58.823723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.823620 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-68drb" Oct 31 03:53:58.972533 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.972495 5490 generic.go:356] "Generic (PLEG): container finished" podID="cdf49e86-d2b1-4888-8475-8a85dd69a6f3" containerID="40c306e87ba45cc24f8f8a23e301f90588e657aafd2470a31afc320248c53bca" exitCode=0 Oct 31 03:53:58.972752 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.972552 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" Oct 31 03:53:58.972752 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.972575 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" event={"ID":"cdf49e86-d2b1-4888-8475-8a85dd69a6f3","Type":"ContainerDied","Data":"40c306e87ba45cc24f8f8a23e301f90588e657aafd2470a31afc320248c53bca"} Oct 31 03:53:58.972752 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.972644 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7" event={"ID":"cdf49e86-d2b1-4888-8475-8a85dd69a6f3","Type":"ContainerDied","Data":"5501d381cbe469ff57e524315ebf0c3609d9f285c811639ef46f62c2fd351c2b"} Oct 31 03:53:58.972752 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.972694 5490 scope.go:122] "RemoveContainer" containerID="40c306e87ba45cc24f8f8a23e301f90588e657aafd2470a31afc320248c53bca" Oct 31 03:53:58.986220 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.986195 5490 scope.go:122] "RemoveContainer" containerID="40c306e87ba45cc24f8f8a23e301f90588e657aafd2470a31afc320248c53bca" Oct 31 03:53:58.986497 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:53:58.986477 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c306e87ba45cc24f8f8a23e301f90588e657aafd2470a31afc320248c53bca\": container with ID starting with 40c306e87ba45cc24f8f8a23e301f90588e657aafd2470a31afc320248c53bca not found: ID does not exist" containerID="40c306e87ba45cc24f8f8a23e301f90588e657aafd2470a31afc320248c53bca" Oct 31 03:53:58.986554 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:58.986507 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c306e87ba45cc24f8f8a23e301f90588e657aafd2470a31afc320248c53bca"} err="failed to get container status \"40c306e87ba45cc24f8f8a23e301f90588e657aafd2470a31afc320248c53bca\": rpc error: code = NotFound desc = could not find container \"40c306e87ba45cc24f8f8a23e301f90588e657aafd2470a31afc320248c53bca\": container with ID starting with 40c306e87ba45cc24f8f8a23e301f90588e657aafd2470a31afc320248c53bca not found: ID does not exist" Oct 31 03:53:59.001422 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:59.001396 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7"] Oct 31 03:53:59.006029 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:59.006004 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-lsmk7"] Oct 31 03:53:59.577830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:53:59.577788 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdf49e86-d2b1-4888-8475-8a85dd69a6f3" path="/var/lib/kubelet/pods/cdf49e86-d2b1-4888-8475-8a85dd69a6f3/volumes" Oct 31 03:54:04.714336 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:04.714298 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wlb6z" Oct 31 03:54:04.780727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:04.778319 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:06.100875 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:06.100646 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:06.853153 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:06.853104 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:07.518152 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:07.516850 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:08.140652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:08.140613 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:08.379995 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:08.378284 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:08.606125 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:08.605290 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:09.343238 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:09.342914 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:10.005845 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:09.994387 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:10.802750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:10.802540 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:11.528704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:11.527759 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:12.273347 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:12.273307 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:12.571137 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:12.571036 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:12.706464 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:12.706423 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:12.856191 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:12.856100 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2"] Oct 31 03:54:12.857548 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:12.857515 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerName="istio-proxy" containerID="cri-o://7cbdd276c928504a14fb06c7051ea8ff32c48ff9af01c3726baa5f4bc21ea05a" gracePeriod=30 Oct 31 03:54:12.921195 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:12.921163 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:13.216913 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:13.216862 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:14.189707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:14.188912 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:14.793709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:14.793659 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:15.574356 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:15.574315 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:15.747457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:15.747421 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-wnnv7" Oct 31 03:54:16.295767 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:16.295720 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:16.901480 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:16.901442 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:17.131817 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:54:17.131779 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:54:17.592114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:17.592074 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:18.031355 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.031314 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:18.154162 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.154130 5490 generic.go:356] "Generic (PLEG): container finished" podID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerID="7cbdd276c928504a14fb06c7051ea8ff32c48ff9af01c3726baa5f4bc21ea05a" exitCode=0 Oct 31 03:54:18.154365 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.154202 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" event={"ID":"cfa0b7ad-1a4c-4803-8214-e61e9f362694","Type":"ContainerDied","Data":"7cbdd276c928504a14fb06c7051ea8ff32c48ff9af01c3726baa5f4bc21ea05a"} Oct 31 03:54:18.154365 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.154274 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" event={"ID":"cfa0b7ad-1a4c-4803-8214-e61e9f362694","Type":"ContainerDied","Data":"5bf06c04d4e177fec8166eaa38cc5726fcb8ce27ccf44d0d0f950455dd0f85d5"} Oct 31 03:54:18.154365 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.154305 5490 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bf06c04d4e177fec8166eaa38cc5726fcb8ce27ccf44d0d0f950455dd0f85d5" Oct 31 03:54:18.154365 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.154325 5490 scope.go:122] "RemoveContainer" containerID="2c93772df1c5e58ab2d9ed2391c07eade76482e50b44d864b1a285aa0c9158fa" Oct 31 03:54:18.165400 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.165374 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:54:18.218704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.210741 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:18.263373 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.263338 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-credential-socket\") pod \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " Oct 31 03:54:18.263571 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.263417 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istiod-ca-cert\") pod \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " Oct 31 03:54:18.263571 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.263543 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-envoy\") pod \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " Oct 31 03:54:18.263571 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.263565 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-data\") pod \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " Oct 31 03:54:18.263911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.263606 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-workload-certs\") pod \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " Oct 31 03:54:18.263911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.263655 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-workload-socket\") pod \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " Oct 31 03:54:18.263911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.263763 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-token\" (UniqueName: \"kubernetes.io/projected/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-token\") pod \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " Oct 31 03:54:18.263911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.263808 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-podinfo\") pod \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " Oct 31 03:54:18.263911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.263862 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa0b7ad-1a4c-4803-8214-e61e9f362694-kube-api-access-vsqtm\" (UniqueName: \"kubernetes.io/projected/cfa0b7ad-1a4c-4803-8214-e61e9f362694-kube-api-access-vsqtm\") pod \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\" (UID: \"cfa0b7ad-1a4c-4803-8214-e61e9f362694\") " Oct 31 03:54:18.264285 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.264251 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-credential-socket" pod "cfa0b7ad-1a4c-4803-8214-e61e9f362694" (UID: "cfa0b7ad-1a4c-4803-8214-e61e9f362694"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:18.264459 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.264433 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-data" pod "cfa0b7ad-1a4c-4803-8214-e61e9f362694" (UID: "cfa0b7ad-1a4c-4803-8214-e61e9f362694"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:18.264542 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.264460 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-workload-socket" pod "cfa0b7ad-1a4c-4803-8214-e61e9f362694" (UID: "cfa0b7ad-1a4c-4803-8214-e61e9f362694"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:18.264687 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.264649 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:18.264775 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.264721 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:18.264775 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.264739 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:18.264890 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.264856 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istiod-ca-cert" pod "cfa0b7ad-1a4c-4803-8214-e61e9f362694" (UID: "cfa0b7ad-1a4c-4803-8214-e61e9f362694"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:18.265177 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.265147 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-workload-certs" pod "cfa0b7ad-1a4c-4803-8214-e61e9f362694" (UID: "cfa0b7ad-1a4c-4803-8214-e61e9f362694"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:18.267535 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.267507 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-envoy" pod "cfa0b7ad-1a4c-4803-8214-e61e9f362694" (UID: "cfa0b7ad-1a4c-4803-8214-e61e9f362694"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:18.267655 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.267549 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-token" pod "cfa0b7ad-1a4c-4803-8214-e61e9f362694" (UID: "cfa0b7ad-1a4c-4803-8214-e61e9f362694"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:18.267655 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.267587 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa0b7ad-1a4c-4803-8214-e61e9f362694-kube-api-access-vsqtm" pod "cfa0b7ad-1a4c-4803-8214-e61e9f362694" (UID: "cfa0b7ad-1a4c-4803-8214-e61e9f362694"). InnerVolumeSpecName "kube-api-access-vsqtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:18.268320 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.268291 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-podinfo" pod "cfa0b7ad-1a4c-4803-8214-e61e9f362694" (UID: "cfa0b7ad-1a4c-4803-8214-e61e9f362694"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:54:18.339033 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.338990 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:18.366729 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.366685 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:18.366729 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.366723 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:18.366729 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.366734 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfa0b7ad-1a4c-4803-8214-e61e9f362694-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:18.366983 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.366743 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:18.366983 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.366753 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfa0b7ad-1a4c-4803-8214-e61e9f362694-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:18.366983 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.366767 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsqtm\" (UniqueName: \"kubernetes.io/projected/cfa0b7ad-1a4c-4803-8214-e61e9f362694-kube-api-access-vsqtm\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:18.825950 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:18.825909 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:19.163823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:19.163793 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2" Oct 31 03:54:19.201598 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:19.199707 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2"] Oct 31 03:54:19.204733 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:19.204705 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-nq5j2"] Oct 31 03:54:19.495063 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:19.494557 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:19.577334 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:19.577299 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" path="/var/lib/kubelet/pods/cfa0b7ad-1a4c-4803-8214-e61e9f362694/volumes" Oct 31 03:54:20.079867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:20.079824 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:20.798505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:20.798460 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:21.399537 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:21.399464 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:21.989727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:21.989623 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:22.741848 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:22.741811 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:23.418664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:23.418614 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:23.721388 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:23.721304 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:23.845742 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:23.845648 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:24.325629 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:24.325576 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:24.903436 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:24.903394 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:25.647656 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:25.647613 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:26.445818 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:26.445731 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:27.209554 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:27.209496 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:27.885731 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:27.885580 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:28.524658 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:28.524612 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:29.202790 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:29.202624 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:29.822002 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:29.821957 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:30.606161 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:30.606062 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:31.217812 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:31.217771 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:31.828259 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:31.828214 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:32.612394 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:32.612146 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:33.243733 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:33.242002 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:33.877359 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:33.877319 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:34.500020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:34.499974 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:35.058822 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:35.058769 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:35.226412 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:35.226324 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:35.376132 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:35.376026 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:35.756645 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:35.756592 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:35.978650 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:35.978602 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:36.519702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:36.518781 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:37.360958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:37.360892 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:37.999423 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:37.998357 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:38.709568 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:38.709520 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:39.607815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:39.607760 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:40.296743 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:40.292213 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:40.932803 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:40.932763 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:41.655364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:41.655322 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:42.096134 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:42.096035 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:42.234825 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:42.234669 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:42.396863 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:42.396757 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn"] Oct 31 03:54:42.397060 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:42.397027 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" podUID="db295e52-5a48-4ec9-a09c-83f8d4b9f62c" containerName="istio-proxy" containerID="cri-o://b386d1b71d98893e74885db42759861c23152f11c16b036cbda9a946f2918c88" gracePeriod=30 Oct 31 03:54:42.402646 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:42.402609 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:42.402907 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:42.402755 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz"] Oct 31 03:54:42.402990 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:42.402965 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" podUID="db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" containerName="istio-proxy" containerID="cri-o://42f35106e7938dfa12cfa21f82be9b5d6f83663d750e29e61ee0d7330a17df1f" gracePeriod=30 Oct 31 03:54:42.652252 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:42.650057 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:43.108386 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.108347 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:43.266112 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.266070 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:43.627879 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.627776 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59"] Oct 31 03:54:43.628844 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.628818 5490 cpu_manager.go:414] "containerMap: removing container" podUID="101e77ad-e8f5-4222-83ee-982ac3954dcc" containerName="istio-proxy" Oct 31 03:54:43.628844 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.628841 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101e77ad-e8f5-4222-83ee-982ac3954dcc" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.628855 5490 cpu_manager.go:414] "containerMap: removing container" podUID="cdf49e86-d2b1-4888-8475-8a85dd69a6f3" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.628861 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdf49e86-d2b1-4888-8475-8a85dd69a6f3" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.628871 5490 cpu_manager.go:414] "containerMap: removing container" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerName="manager" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.628876 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerName="manager" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.628886 5490 cpu_manager.go:414] "containerMap: removing container" podUID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.628890 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.628898 5490 cpu_manager.go:414] "containerMap: removing container" podUID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.628903 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.628914 5490 cpu_manager.go:414] "containerMap: removing container" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.628918 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.628926 5490 cpu_manager.go:414] "containerMap: removing container" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.628931 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.628944 5490 cpu_manager.go:414] "containerMap: removing container" podUID="dcfb42f0-a328-4179-9a5d-ac6669e132a8" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.628951 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcfb42f0-a328-4179-9a5d-ac6669e132a8" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.628973 5490 cpu_manager.go:414] "containerMap: removing container" podUID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.628980 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.628998 5490 cpu_manager.go:414] "containerMap: removing container" podUID="5123bf1e-0ad6-4406-86b0-2b9a9e025a45" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629005 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5123bf1e-0ad6-4406-86b0-2b9a9e025a45" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629018 5490 cpu_manager.go:414] "containerMap: removing container" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerName="manager" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629025 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerName="manager" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629055 5490 cpu_manager.go:414] "containerMap: removing container" podUID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629063 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629075 5490 cpu_manager.go:414] "containerMap: removing container" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629082 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629285 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcfb42f0-a328-4179-9a5d-ac6669e132a8" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629302 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629312 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629326 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="5123bf1e-0ad6-4406-86b0-2b9a9e025a45" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629337 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerName="manager" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629349 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629362 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef5f9523-ec4a-4372-a327-009c9fc40563" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629373 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c13f49-3afa-4080-bcaf-e9fcba5f5fc7" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629385 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="101e77ad-e8f5-4222-83ee-982ac3954dcc" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629391 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdf49e86-d2b1-4888-8475-8a85dd69a6f3" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629397 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0f0e9f6-2e01-4338-8243-346ffb2c7cc2" containerName="manager" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629404 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629414 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e563544-8d00-4f0f-bd8a-c0cfc2c5c666" containerName="istio-proxy" Oct 31 03:54:43.630116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.629423 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerName="istio-proxy" Oct 31 03:54:43.634994 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.634971 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.642976 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.642932 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-20585c-openshift-default-dockercfg-mnxkq\"" Oct 31 03:54:43.652003 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.651970 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59"] Oct 31 03:54:43.716416 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.716359 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:43.772700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.769851 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-envoy\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.772700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.770055 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-podinfo\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.772700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.770091 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-credential-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.772700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.770125 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-token\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.772700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.770175 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-workload-certs\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.772700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.770214 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istiod-ca-cert\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.772700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.770249 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-workload-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.772700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.770364 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hd5l\" (UniqueName: \"kubernetes.io/projected/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-kube-api-access-6hd5l\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.772700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.770396 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-data\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.871579 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.871539 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hd5l\" (UniqueName: \"kubernetes.io/projected/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-kube-api-access-6hd5l\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.871810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.871594 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-data\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.871810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.871689 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-envoy\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.871959 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.871878 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-podinfo\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.871959 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.871916 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-credential-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.872076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.871968 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-token\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.872076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.872034 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-workload-certs\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.872186 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.872085 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istiod-ca-cert\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.872186 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.872131 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-workload-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.872618 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.872082 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-data\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.872618 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.872546 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-workload-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.872618 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.872571 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-credential-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.872866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.872839 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-workload-certs\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.872956 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.872934 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istiod-ca-cert\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.876180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.876154 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-envoy\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.876510 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.876483 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-podinfo\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.880465 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.880396 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-token\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.880465 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.880410 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hd5l\" (UniqueName: \"kubernetes.io/projected/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-kube-api-access-6hd5l\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:43.914834 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.914710 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:43.964810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:43.964769 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:44.100217 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:44.100171 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:44.278603 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:54:44.278571 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a03b21_4c4f_422e_baa4_6e905b1b0bf5.slice/crio-5507ec24be6b2f8a27902bcaf56e493e87f883e1db69744fb2b2bbf5632c0af6 WatchSource:0}: Error finding container 5507ec24be6b2f8a27902bcaf56e493e87f883e1db69744fb2b2bbf5632c0af6: Status 404 returned error can't find the container with id 5507ec24be6b2f8a27902bcaf56e493e87f883e1db69744fb2b2bbf5632c0af6 Oct 31 03:54:44.279403 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:44.279376 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59"] Oct 31 03:54:44.282244 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:44.282196 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:54:44.282367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:44.282284 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:54:44.282367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:44.282330 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:54:44.447701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:44.447115 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" event={"ID":"24a03b21-4c4f-422e-baa4-6e905b1b0bf5","Type":"ContainerStarted","Data":"770d923a22550d4a90ab065ee3eb02df2ca0b1192f8fe0fbe642e55baff56e6b"} Oct 31 03:54:44.447701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:44.447187 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" event={"ID":"24a03b21-4c4f-422e-baa4-6e905b1b0bf5","Type":"ContainerStarted","Data":"5507ec24be6b2f8a27902bcaf56e493e87f883e1db69744fb2b2bbf5632c0af6"} Oct 31 03:54:44.480090 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:44.480027 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" podStartSLOduration=1.480011887 podStartE2EDuration="1.480011887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:44.477410747 +0000 UTC m=+2319.012105703" watchObservedRunningTime="2026-10-31 03:54:44.480011887 +0000 UTC m=+2319.014706842" Oct 31 03:54:44.833340 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:44.833284 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:44.965378 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:44.965328 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:45.891481 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:45.890726 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:46.046703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:46.038011 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:46.489505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:46.489464 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:46.494704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:46.493311 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:54:47.130414 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.130357 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:47.137830 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:54:47.137794 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:54:47.337713 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.331226 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:47.480868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.480793 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-s9sn6"] Oct 31 03:54:47.483309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.482023 5490 cpu_manager.go:414] "containerMap: removing container" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" Oct 31 03:54:47.483309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.482062 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" Oct 31 03:54:47.483309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.482076 5490 cpu_manager.go:414] "containerMap: removing container" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerName="istio-proxy" Oct 31 03:54:47.483309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.482085 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa0b7ad-1a4c-4803-8214-e61e9f362694" containerName="istio-proxy" Oct 31 03:54:47.483309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.482420 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2dbe7ba-df3f-4b3f-a6c8-6e386172702b" containerName="istio-proxy" Oct 31 03:54:47.500885 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.495096 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-s9sn6" Oct 31 03:54:47.501481 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.501416 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 31 03:54:47.501481 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.501440 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 31 03:54:47.501800 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.501784 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-xgnxw\"" Oct 31 03:54:47.508843 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.505489 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-s9sn6"] Oct 31 03:54:47.607624 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.604883 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce9fae9e-74e8-4406-bc98-fd4ba1a319bd-cert\") pod \"odh-model-controller-769875db56-s9sn6\" (UID: \"ce9fae9e-74e8-4406-bc98-fd4ba1a319bd\") " pod="opendatahub/odh-model-controller-769875db56-s9sn6" Oct 31 03:54:47.607624 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.605022 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce9fae9e-74e8-4406-bc98-fd4ba1a319bd-metrics-certs\") pod \"odh-model-controller-769875db56-s9sn6\" (UID: \"ce9fae9e-74e8-4406-bc98-fd4ba1a319bd\") " pod="opendatahub/odh-model-controller-769875db56-s9sn6" Oct 31 03:54:47.607624 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.605363 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2mqw\" (UniqueName: \"kubernetes.io/projected/ce9fae9e-74e8-4406-bc98-fd4ba1a319bd-kube-api-access-q2mqw\") pod \"odh-model-controller-769875db56-s9sn6\" (UID: \"ce9fae9e-74e8-4406-bc98-fd4ba1a319bd\") " pod="opendatahub/odh-model-controller-769875db56-s9sn6" Oct 31 03:54:47.711708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.711589 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce9fae9e-74e8-4406-bc98-fd4ba1a319bd-cert\") pod \"odh-model-controller-769875db56-s9sn6\" (UID: \"ce9fae9e-74e8-4406-bc98-fd4ba1a319bd\") " pod="opendatahub/odh-model-controller-769875db56-s9sn6" Oct 31 03:54:47.711896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.711723 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce9fae9e-74e8-4406-bc98-fd4ba1a319bd-metrics-certs\") pod \"odh-model-controller-769875db56-s9sn6\" (UID: \"ce9fae9e-74e8-4406-bc98-fd4ba1a319bd\") " pod="opendatahub/odh-model-controller-769875db56-s9sn6" Oct 31 03:54:47.712034 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.711985 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2mqw\" (UniqueName: \"kubernetes.io/projected/ce9fae9e-74e8-4406-bc98-fd4ba1a319bd-kube-api-access-q2mqw\") pod \"odh-model-controller-769875db56-s9sn6\" (UID: \"ce9fae9e-74e8-4406-bc98-fd4ba1a319bd\") " pod="opendatahub/odh-model-controller-769875db56-s9sn6" Oct 31 03:54:47.716720 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.716670 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce9fae9e-74e8-4406-bc98-fd4ba1a319bd-cert\") pod \"odh-model-controller-769875db56-s9sn6\" (UID: \"ce9fae9e-74e8-4406-bc98-fd4ba1a319bd\") " pod="opendatahub/odh-model-controller-769875db56-s9sn6" Oct 31 03:54:47.720224 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.720186 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce9fae9e-74e8-4406-bc98-fd4ba1a319bd-metrics-certs\") pod \"odh-model-controller-769875db56-s9sn6\" (UID: \"ce9fae9e-74e8-4406-bc98-fd4ba1a319bd\") " pod="opendatahub/odh-model-controller-769875db56-s9sn6" Oct 31 03:54:47.721995 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.721968 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2mqw\" (UniqueName: \"kubernetes.io/projected/ce9fae9e-74e8-4406-bc98-fd4ba1a319bd-kube-api-access-q2mqw\") pod \"odh-model-controller-769875db56-s9sn6\" (UID: \"ce9fae9e-74e8-4406-bc98-fd4ba1a319bd\") " pod="opendatahub/odh-model-controller-769875db56-s9sn6" Oct 31 03:54:47.807063 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.807030 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:54:47.884861 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.884574 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-s9sn6" Oct 31 03:54:47.901276 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.894279 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:54:47.916237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.916063 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-kube-api-access-qr7lb\" (UniqueName: \"kubernetes.io/projected/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-kube-api-access-qr7lb\") pod \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " Oct 31 03:54:47.916237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.916196 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istiod-ca-cert\") pod \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " Oct 31 03:54:47.916237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.916234 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-credential-socket\") pod \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " Oct 31 03:54:47.916479 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.916335 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-workload-socket\") pod \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " Oct 31 03:54:47.916479 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.916384 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-podinfo\") pod \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " Oct 31 03:54:47.916479 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.916460 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-workload-certs\") pod \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " Oct 31 03:54:47.916635 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.916576 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-token\" (UniqueName: \"kubernetes.io/projected/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-token\") pod \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " Oct 31 03:54:47.916704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.916607 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istiod-ca-cert" pod "db295e52-5a48-4ec9-a09c-83f8d4b9f62c" (UID: "db295e52-5a48-4ec9-a09c-83f8d4b9f62c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:47.916761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.916704 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-data\") pod \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " Oct 31 03:54:47.916803 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.916783 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-envoy\") pod \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\" (UID: \"db295e52-5a48-4ec9-a09c-83f8d4b9f62c\") " Oct 31 03:54:47.916839 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.916811 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-workload-socket" pod "db295e52-5a48-4ec9-a09c-83f8d4b9f62c" (UID: "db295e52-5a48-4ec9-a09c-83f8d4b9f62c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:47.917244 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.916958 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-credential-socket" pod "db295e52-5a48-4ec9-a09c-83f8d4b9f62c" (UID: "db295e52-5a48-4ec9-a09c-83f8d4b9f62c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:47.917244 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.917084 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-workload-certs" pod "db295e52-5a48-4ec9-a09c-83f8d4b9f62c" (UID: "db295e52-5a48-4ec9-a09c-83f8d4b9f62c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:47.917432 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.917326 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-data" pod "db295e52-5a48-4ec9-a09c-83f8d4b9f62c" (UID: "db295e52-5a48-4ec9-a09c-83f8d4b9f62c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:47.917573 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.917542 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:47.917626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.917571 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:47.917626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.917587 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:47.917626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.917601 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:47.917626 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.917613 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:47.920340 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.920260 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-podinfo" pod "db295e52-5a48-4ec9-a09c-83f8d4b9f62c" (UID: "db295e52-5a48-4ec9-a09c-83f8d4b9f62c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:54:47.920606 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.920569 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-token" pod "db295e52-5a48-4ec9-a09c-83f8d4b9f62c" (UID: "db295e52-5a48-4ec9-a09c-83f8d4b9f62c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:47.921769 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.921735 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-envoy" pod "db295e52-5a48-4ec9-a09c-83f8d4b9f62c" (UID: "db295e52-5a48-4ec9-a09c-83f8d4b9f62c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:47.939691 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:47.939620 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-kube-api-access-qr7lb" pod "db295e52-5a48-4ec9-a09c-83f8d4b9f62c" (UID: "db295e52-5a48-4ec9-a09c-83f8d4b9f62c"). InnerVolumeSpecName "kube-api-access-qr7lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:48.018663 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.018627 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-workload-socket\") pod \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " Oct 31 03:54:48.018855 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.018734 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-credential-socket\") pod \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " Oct 31 03:54:48.018855 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.018804 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-envoy\") pod \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " Oct 31 03:54:48.018855 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.018850 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-token\" (UniqueName: \"kubernetes.io/projected/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-token\") pod \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " Oct 31 03:54:48.019013 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.018945 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-kube-api-access-xjl9q\" (UniqueName: \"kubernetes.io/projected/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-kube-api-access-xjl9q\") pod \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " Oct 31 03:54:48.019073 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.019033 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-data\") pod \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " Oct 31 03:54:48.020654 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.019174 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istiod-ca-cert\") pod \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " Oct 31 03:54:48.020654 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.019225 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-podinfo\") pod \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " Oct 31 03:54:48.020654 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.019283 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-workload-certs\") pod \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\" (UID: \"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5\") " Oct 31 03:54:48.020654 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.020367 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-workload-socket" pod "db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" (UID: "db4dcf84-a219-4ff2-b4ff-5549ef2fafe5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:48.020654 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.020622 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-credential-socket" pod "db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" (UID: "db4dcf84-a219-4ff2-b4ff-5549ef2fafe5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:48.022932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.022105 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-data" pod "db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" (UID: "db4dcf84-a219-4ff2-b4ff-5549ef2fafe5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:48.022932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.022304 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istiod-ca-cert" pod "db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" (UID: "db4dcf84-a219-4ff2-b4ff-5549ef2fafe5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:54:48.022932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.022898 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-workload-certs" pod "db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" (UID: "db4dcf84-a219-4ff2-b4ff-5549ef2fafe5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:48.024404 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.024239 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.024404 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.024269 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.024404 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.024285 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.024404 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.024300 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.024404 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.024314 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.024404 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.024329 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.024404 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.024344 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.024404 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.024363 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr7lb\" (UniqueName: \"kubernetes.io/projected/db295e52-5a48-4ec9-a09c-83f8d4b9f62c-kube-api-access-qr7lb\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.024404 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.024379 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.025236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.025196 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-envoy" pod "db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" (UID: "db4dcf84-a219-4ff2-b4ff-5549ef2fafe5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:54:48.027913 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.027877 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-podinfo" pod "db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" (UID: "db4dcf84-a219-4ff2-b4ff-5549ef2fafe5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:54:48.028164 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.028133 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-token" pod "db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" (UID: "db4dcf84-a219-4ff2-b4ff-5549ef2fafe5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:48.028519 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.028496 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-kube-api-access-xjl9q" pod "db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" (UID: "db4dcf84-a219-4ff2-b4ff-5549ef2fafe5"). InnerVolumeSpecName "kube-api-access-xjl9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:54:48.126246 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.126200 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.126246 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.126248 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.126483 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.126264 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjl9q\" (UniqueName: \"kubernetes.io/projected/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-kube-api-access-xjl9q\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.126483 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.126280 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:54:48.239292 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:54:48.239260 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9fae9e_74e8_4406_bc98_fd4ba1a319bd.slice/crio-3c11232eee4c2725dd808d667494df22a20faa1a01aa7f8184168e5b9510c8e7 WatchSource:0}: Error finding container 3c11232eee4c2725dd808d667494df22a20faa1a01aa7f8184168e5b9510c8e7: Status 404 returned error can't find the container with id 3c11232eee4c2725dd808d667494df22a20faa1a01aa7f8184168e5b9510c8e7 Oct 31 03:54:48.240809 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.240780 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-s9sn6"] Oct 31 03:54:48.335326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.335233 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:48.512204 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.512165 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-s9sn6" event={"ID":"ce9fae9e-74e8-4406-bc98-fd4ba1a319bd","Type":"ContainerStarted","Data":"3c11232eee4c2725dd808d667494df22a20faa1a01aa7f8184168e5b9510c8e7"} Oct 31 03:54:48.513887 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.513855 5490 generic.go:356] "Generic (PLEG): container finished" podID="db295e52-5a48-4ec9-a09c-83f8d4b9f62c" containerID="b386d1b71d98893e74885db42759861c23152f11c16b036cbda9a946f2918c88" exitCode=0 Oct 31 03:54:48.514042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.513913 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" Oct 31 03:54:48.514042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.513936 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" event={"ID":"db295e52-5a48-4ec9-a09c-83f8d4b9f62c","Type":"ContainerDied","Data":"b386d1b71d98893e74885db42759861c23152f11c16b036cbda9a946f2918c88"} Oct 31 03:54:48.514042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.513994 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn" event={"ID":"db295e52-5a48-4ec9-a09c-83f8d4b9f62c","Type":"ContainerDied","Data":"2f6b3d00b62c4b05496eb21f1cbf6581f9ccce68c0531c5278f551733f67d45d"} Oct 31 03:54:48.514042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.514025 5490 scope.go:122] "RemoveContainer" containerID="b386d1b71d98893e74885db42759861c23152f11c16b036cbda9a946f2918c88" Oct 31 03:54:48.517350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.517141 5490 generic.go:356] "Generic (PLEG): container finished" podID="db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" containerID="42f35106e7938dfa12cfa21f82be9b5d6f83663d750e29e61ee0d7330a17df1f" exitCode=0 Oct 31 03:54:48.517350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.517194 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" event={"ID":"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5","Type":"ContainerDied","Data":"42f35106e7938dfa12cfa21f82be9b5d6f83663d750e29e61ee0d7330a17df1f"} Oct 31 03:54:48.517350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.517208 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" Oct 31 03:54:48.517350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.517247 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz" event={"ID":"db4dcf84-a219-4ff2-b4ff-5549ef2fafe5","Type":"ContainerDied","Data":"86d75959bc8a9bc980cb0da80bbe2b79ab98586affa76142c70b4ebdf78aae69"} Oct 31 03:54:48.532542 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.532514 5490 scope.go:122] "RemoveContainer" containerID="b386d1b71d98893e74885db42759861c23152f11c16b036cbda9a946f2918c88" Oct 31 03:54:48.532901 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:54:48.532870 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b386d1b71d98893e74885db42759861c23152f11c16b036cbda9a946f2918c88\": container with ID starting with b386d1b71d98893e74885db42759861c23152f11c16b036cbda9a946f2918c88 not found: ID does not exist" containerID="b386d1b71d98893e74885db42759861c23152f11c16b036cbda9a946f2918c88" Oct 31 03:54:48.532982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.532913 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b386d1b71d98893e74885db42759861c23152f11c16b036cbda9a946f2918c88"} err="failed to get container status \"b386d1b71d98893e74885db42759861c23152f11c16b036cbda9a946f2918c88\": rpc error: code = NotFound desc = could not find container \"b386d1b71d98893e74885db42759861c23152f11c16b036cbda9a946f2918c88\": container with ID starting with b386d1b71d98893e74885db42759861c23152f11c16b036cbda9a946f2918c88 not found: ID does not exist" Oct 31 03:54:48.532982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.532933 5490 scope.go:122] "RemoveContainer" containerID="42f35106e7938dfa12cfa21f82be9b5d6f83663d750e29e61ee0d7330a17df1f" Oct 31 03:54:48.545445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.545412 5490 scope.go:122] "RemoveContainer" containerID="42f35106e7938dfa12cfa21f82be9b5d6f83663d750e29e61ee0d7330a17df1f" Oct 31 03:54:48.545826 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:54:48.545796 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42f35106e7938dfa12cfa21f82be9b5d6f83663d750e29e61ee0d7330a17df1f\": container with ID starting with 42f35106e7938dfa12cfa21f82be9b5d6f83663d750e29e61ee0d7330a17df1f not found: ID does not exist" containerID="42f35106e7938dfa12cfa21f82be9b5d6f83663d750e29e61ee0d7330a17df1f" Oct 31 03:54:48.545928 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.545841 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42f35106e7938dfa12cfa21f82be9b5d6f83663d750e29e61ee0d7330a17df1f"} err="failed to get container status \"42f35106e7938dfa12cfa21f82be9b5d6f83663d750e29e61ee0d7330a17df1f\": rpc error: code = NotFound desc = could not find container \"42f35106e7938dfa12cfa21f82be9b5d6f83663d750e29e61ee0d7330a17df1f\": container with ID starting with 42f35106e7938dfa12cfa21f82be9b5d6f83663d750e29e61ee0d7330a17df1f not found: ID does not exist" Oct 31 03:54:48.566194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.566163 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn"] Oct 31 03:54:48.573930 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.573903 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-5tvwn"] Oct 31 03:54:48.600607 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.600520 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz"] Oct 31 03:54:48.606733 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.606699 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-8plfz"] Oct 31 03:54:48.825019 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.824978 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:48.826496 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.826469 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5"] Oct 31 03:54:48.827395 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.827369 5490 cpu_manager.go:414] "containerMap: removing container" podUID="db295e52-5a48-4ec9-a09c-83f8d4b9f62c" containerName="istio-proxy" Oct 31 03:54:48.827395 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.827394 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db295e52-5a48-4ec9-a09c-83f8d4b9f62c" containerName="istio-proxy" Oct 31 03:54:48.827587 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.827421 5490 cpu_manager.go:414] "containerMap: removing container" podUID="db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" containerName="istio-proxy" Oct 31 03:54:48.827587 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.827429 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" containerName="istio-proxy" Oct 31 03:54:48.827707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.827646 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="db295e52-5a48-4ec9-a09c-83f8d4b9f62c" containerName="istio-proxy" Oct 31 03:54:48.827707 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.827666 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" containerName="istio-proxy" Oct 31 03:54:48.832992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.832970 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5" Oct 31 03:54:48.840779 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.840754 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-20585c\"" Oct 31 03:54:48.848116 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.847268 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5"] Oct 31 03:54:48.867788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.865877 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq"] Oct 31 03:54:48.878790 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.878764 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq" Oct 31 03:54:48.888693 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.886385 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-20585c-dockercfg-w6rvg\"" Oct 31 03:54:48.915477 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.915445 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq"] Oct 31 03:54:48.952420 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.952108 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq\" (UID: \"679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq" Oct 31 03:54:48.952420 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.952173 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4d6w\" (UniqueName: \"kubernetes.io/projected/679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2-kube-api-access-m4d6w\") pod \"payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq\" (UID: \"679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq" Oct 31 03:54:48.953332 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.953202 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ckr4\" (UniqueName: \"kubernetes.io/projected/2cf184db-500a-41eb-8ca5-7a40b2645693-kube-api-access-7ckr4\") pod \"payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5\" (UID: \"2cf184db-500a-41eb-8ca5-7a40b2645693\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5" Oct 31 03:54:48.953967 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:48.953698 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cf184db-500a-41eb-8ca5-7a40b2645693-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5\" (UID: \"2cf184db-500a-41eb-8ca5-7a40b2645693\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5" Oct 31 03:54:49.007730 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:49.007377 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:49.152919 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:49.152580 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ckr4\" (UniqueName: \"kubernetes.io/projected/2cf184db-500a-41eb-8ca5-7a40b2645693-kube-api-access-7ckr4\") pod \"payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5\" (UID: \"2cf184db-500a-41eb-8ca5-7a40b2645693\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5" Oct 31 03:54:49.152919 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:49.152758 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cf184db-500a-41eb-8ca5-7a40b2645693-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5\" (UID: \"2cf184db-500a-41eb-8ca5-7a40b2645693\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5" Oct 31 03:54:49.172700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:49.160581 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq\" (UID: \"679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq" Oct 31 03:54:49.172700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:49.160650 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4d6w\" (UniqueName: \"kubernetes.io/projected/679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2-kube-api-access-m4d6w\") pod \"payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq\" (UID: \"679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq" Oct 31 03:54:49.174053 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:49.173586 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cf184db-500a-41eb-8ca5-7a40b2645693-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5\" (UID: \"2cf184db-500a-41eb-8ca5-7a40b2645693\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5" Oct 31 03:54:49.174252 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:49.173885 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq\" (UID: \"679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq" Oct 31 03:54:49.201917 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:49.188194 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ckr4\" (UniqueName: \"kubernetes.io/projected/2cf184db-500a-41eb-8ca5-7a40b2645693-kube-api-access-7ckr4\") pod \"payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5\" (UID: \"2cf184db-500a-41eb-8ca5-7a40b2645693\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5" Oct 31 03:54:49.208302 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:49.208267 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4d6w\" (UniqueName: \"kubernetes.io/projected/679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2-kube-api-access-m4d6w\") pod \"payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq\" (UID: \"679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq" Oct 31 03:54:49.208861 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:49.208573 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5" Oct 31 03:54:49.241955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:49.241898 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq" Oct 31 03:54:49.520594 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:49.520563 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5"] Oct 31 03:54:49.532248 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:49.532217 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-s9sn6" event={"ID":"ce9fae9e-74e8-4406-bc98-fd4ba1a319bd","Type":"ContainerStarted","Data":"9f61ec1d000d5cec2dea92ca71fe407d291565c7a3dc2186839f07b07d21ef8e"} Oct 31 03:54:49.532831 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:49.532473 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-s9sn6" Oct 31 03:54:49.556316 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:49.556258 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-s9sn6" podStartSLOduration=1.9896426809999999 podStartE2EDuration="2.556212205s" totalImagesPullingTime="566.569524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:49.553844835 +0000 UTC m=+2324.088539791" watchObservedRunningTime="2026-10-31 03:54:49.556212205 +0000 UTC m=+2324.090907160" Oct 31 03:54:49.580760 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:49.580716 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db295e52-5a48-4ec9-a09c-83f8d4b9f62c" path="/var/lib/kubelet/pods/db295e52-5a48-4ec9-a09c-83f8d4b9f62c/volumes" Oct 31 03:54:49.582202 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:49.582175 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db4dcf84-a219-4ff2-b4ff-5549ef2fafe5" path="/var/lib/kubelet/pods/db4dcf84-a219-4ff2-b4ff-5549ef2fafe5/volumes" Oct 31 03:54:49.586954 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:49.586924 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq"] Oct 31 03:54:49.587703 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:54:49.587667 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679e88b7_c835_4f3b_bb3b_4aa5a1fcefe2.slice/crio-5daec0a94d37c5e5a88956ffa41668ff18c9e24bd8bc39db874cd4d01737221f WatchSource:0}: Error finding container 5daec0a94d37c5e5a88956ffa41668ff18c9e24bd8bc39db874cd4d01737221f: Status 404 returned error can't find the container with id 5daec0a94d37c5e5a88956ffa41668ff18c9e24bd8bc39db874cd4d01737221f Oct 31 03:54:49.599820 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:49.599789 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:49.627923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:49.627890 5490 scope.go:122] "RemoveContainer" containerID="7cbdd276c928504a14fb06c7051ea8ff32c48ff9af01c3726baa5f4bc21ea05a" Oct 31 03:54:50.545364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:50.545325 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq" event={"ID":"679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2","Type":"ContainerStarted","Data":"ed6d1bdf222e1a94cde8ab9102d20da712aa9cec10da51754d6641048c8c4ca8"} Oct 31 03:54:50.545882 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:50.545417 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq" event={"ID":"679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2","Type":"ContainerStarted","Data":"5daec0a94d37c5e5a88956ffa41668ff18c9e24bd8bc39db874cd4d01737221f"} Oct 31 03:54:50.545882 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:50.545730 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq" Oct 31 03:54:50.547407 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:50.547381 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5" event={"ID":"2cf184db-500a-41eb-8ca5-7a40b2645693","Type":"ContainerStarted","Data":"1ed37b99793e18547ecfef0a6931a7ab57f9c50ce6f0b6efbb24bade256fc997"} Oct 31 03:54:50.547573 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:50.547425 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5" event={"ID":"2cf184db-500a-41eb-8ca5-7a40b2645693","Type":"ContainerStarted","Data":"63fcabdf2c16dab93ef2fdc1bc2c560b4209d655e6cc00858ceb0ca8bbbc54f2"} Oct 31 03:54:50.547573 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:50.547530 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5" Oct 31 03:54:50.564146 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:50.564089 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq" podStartSLOduration=2.5640734419999998 podStartE2EDuration="2.564073442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:50.562329163 +0000 UTC m=+2325.097024119" watchObservedRunningTime="2026-10-31 03:54:50.564073442 +0000 UTC m=+2325.098768438" Oct 31 03:54:50.586772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:50.586714 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5" podStartSLOduration=2.5866955149999997 podStartE2EDuration="2.586695515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:50.583224224 +0000 UTC m=+2325.117919184" watchObservedRunningTime="2026-10-31 03:54:50.586695515 +0000 UTC m=+2325.121390470" Oct 31 03:54:50.623456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:50.623417 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:51.271820 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:51.271778 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:51.910103 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:51.910062 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:52.706822 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:52.705761 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:53.390103 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:53.388301 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:54.169453 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:54.169416 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:54.917212 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:54.917175 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:55.524229 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:55.524182 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:56.226524 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:56.226472 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:56.925532 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:56.925439 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:57.798145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:57.798100 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:58.409790 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:58.409503 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:59.319432 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:59.319389 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:54:59.994727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:54:59.994648 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:00.550333 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:00.550290 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-s9sn6" Oct 31 03:55:00.615755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:00.615665 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:01.216814 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:01.216767 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:01.558838 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:01.558755 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5" Oct 31 03:55:01.558838 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:01.558827 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq" Oct 31 03:55:01.917094 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:01.917049 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:02.588291 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:02.588251 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:03.178413 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:03.177645 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:03.850121 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:03.832334 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:04.396003 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:04.395967 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb"] Oct 31 03:55:04.401652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:04.401627 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" Oct 31 03:55:04.410090 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:04.410060 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb"] Oct 31 03:55:04.410329 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:04.410307 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-20585c-serving-cert\"" Oct 31 03:55:04.495867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:04.495832 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:04.557891 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:04.557848 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d55416a-52be-41b0-aef9-7709346f1fdd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb\" (UID: \"7d55416a-52be-41b0-aef9-7709346f1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" Oct 31 03:55:04.558115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:04.558098 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlctl\" (UniqueName: \"kubernetes.io/projected/7d55416a-52be-41b0-aef9-7709346f1fdd-kube-api-access-dlctl\") pod \"maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb\" (UID: \"7d55416a-52be-41b0-aef9-7709346f1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" Oct 31 03:55:04.558291 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:04.558270 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d55416a-52be-41b0-aef9-7709346f1fdd-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb\" (UID: \"7d55416a-52be-41b0-aef9-7709346f1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" Oct 31 03:55:04.558356 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:04.558317 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d55416a-52be-41b0-aef9-7709346f1fdd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb\" (UID: \"7d55416a-52be-41b0-aef9-7709346f1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" Oct 31 03:55:04.626329 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:04.626285 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:04.660000 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:04.659964 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlctl\" (UniqueName: \"kubernetes.io/projected/7d55416a-52be-41b0-aef9-7709346f1fdd-kube-api-access-dlctl\") pod \"maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb\" (UID: \"7d55416a-52be-41b0-aef9-7709346f1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" Oct 31 03:55:04.660184 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:04.660100 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d55416a-52be-41b0-aef9-7709346f1fdd-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb\" (UID: \"7d55416a-52be-41b0-aef9-7709346f1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" Oct 31 03:55:04.660184 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:04.660135 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d55416a-52be-41b0-aef9-7709346f1fdd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb\" (UID: \"7d55416a-52be-41b0-aef9-7709346f1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" Oct 31 03:55:04.660274 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:04.660186 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d55416a-52be-41b0-aef9-7709346f1fdd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb\" (UID: \"7d55416a-52be-41b0-aef9-7709346f1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" Oct 31 03:55:04.660967 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:04.660916 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d55416a-52be-41b0-aef9-7709346f1fdd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb\" (UID: \"7d55416a-52be-41b0-aef9-7709346f1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" Oct 31 03:55:04.663285 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:04.663256 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d55416a-52be-41b0-aef9-7709346f1fdd-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb\" (UID: \"7d55416a-52be-41b0-aef9-7709346f1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" Oct 31 03:55:04.663384 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:04.663265 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d55416a-52be-41b0-aef9-7709346f1fdd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb\" (UID: \"7d55416a-52be-41b0-aef9-7709346f1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" Oct 31 03:55:04.668665 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:04.668642 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlctl\" (UniqueName: \"kubernetes.io/projected/7d55416a-52be-41b0-aef9-7709346f1fdd-kube-api-access-dlctl\") pod \"maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb\" (UID: \"7d55416a-52be-41b0-aef9-7709346f1fdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" Oct 31 03:55:04.726468 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:04.726433 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" Oct 31 03:55:05.040157 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:05.039217 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb"] Oct 31 03:55:05.044192 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:55:05.044142 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d55416a_52be_41b0_aef9_7709346f1fdd.slice/crio-ca01093eedb903f00d9ebaec9ba2499969c11c515ae79c69fd5f799f55f6b4e6 WatchSource:0}: Error finding container ca01093eedb903f00d9ebaec9ba2499969c11c515ae79c69fd5f799f55f6b4e6: Status 404 returned error can't find the container with id ca01093eedb903f00d9ebaec9ba2499969c11c515ae79c69fd5f799f55f6b4e6 Oct 31 03:55:05.223625 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:05.223572 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:05.699698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:05.682601 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" event={"ID":"7d55416a-52be-41b0-aef9-7709346f1fdd","Type":"ContainerStarted","Data":"54f575127787e893c81f50bae1cffc63850e0622cd625e4dd188a748b0a6066d"} Oct 31 03:55:05.699698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:05.682666 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" event={"ID":"7d55416a-52be-41b0-aef9-7709346f1fdd","Type":"ContainerStarted","Data":"ca01093eedb903f00d9ebaec9ba2499969c11c515ae79c69fd5f799f55f6b4e6"} Oct 31 03:55:05.699698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:05.696360 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" Oct 31 03:55:05.720731 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:05.719113 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" podStartSLOduration=1.190120628 podStartE2EDuration="1.719098559s" totalImagesPullingTime="528.977931ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:05.718459654 +0000 UTC m=+2340.253154610" watchObservedRunningTime="2026-10-31 03:55:05.719098559 +0000 UTC m=+2340.253793514" Oct 31 03:55:05.905789 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:05.905533 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:06.656863 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:06.656821 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:07.673063 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:07.673028 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:08.406523 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:08.406488 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:09.400175 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:09.400140 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:10.367312 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:10.367265 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:11.108154 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:11.108103 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:11.902698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:11.899587 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:12.251456 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.251419 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf"] Oct 31 03:55:12.256162 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.256130 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.267393 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.266858 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a5d9b6-openshift-default-dockercfg-fjkqj\"" Oct 31 03:55:12.269572 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.269540 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf"] Oct 31 03:55:12.276046 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.276016 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45"] Oct 31 03:55:12.282316 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.282267 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.290635 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.290005 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45"] Oct 31 03:55:12.323040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.318400 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.323040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.318503 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.323040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.318745 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.323040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.318802 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.323040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.318865 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.323040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.318945 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7snbb\" (UniqueName: \"kubernetes.io/projected/01e9fcdd-b435-47dc-a064-02ff1d074156-kube-api-access-7snbb\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.323040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.319050 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01e9fcdd-b435-47dc-a064-02ff1d074156-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.323040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.319073 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.323040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.319100 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.420920 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.420876 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.421135 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.420944 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.421135 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.421006 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.421135 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.421065 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7snbb\" (UniqueName: \"kubernetes.io/projected/01e9fcdd-b435-47dc-a064-02ff1d074156-kube-api-access-7snbb\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.421135 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.421098 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.421338 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.421177 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smhgj\" (UniqueName: \"kubernetes.io/projected/baea2599-5723-4c5e-b063-e960ec09e121-kube-api-access-smhgj\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.421338 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.421308 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.421338 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.421312 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.421460 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.421375 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.421460 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.421388 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01e9fcdd-b435-47dc-a064-02ff1d074156-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.421460 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.421431 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.421460 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.421442 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.421603 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.421486 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.421603 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.421543 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.421603 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.421564 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.421785 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.421609 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baea2599-5723-4c5e-b063-e960ec09e121-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.421785 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.421641 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baea2599-5723-4c5e-b063-e960ec09e121-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.421785 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.421757 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.422590 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.421831 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.422590 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.421884 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.422590 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.421965 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.422590 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.422024 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baea2599-5723-4c5e-b063-e960ec09e121-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.422590 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.422055 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01e9fcdd-b435-47dc-a064-02ff1d074156-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.423954 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.423932 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.424070 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.424053 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.430345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.430101 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.430345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.430296 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7snbb\" (UniqueName: \"kubernetes.io/projected/01e9fcdd-b435-47dc-a064-02ff1d074156-kube-api-access-7snbb\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.523384 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.523297 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.523384 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.523339 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.523384 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.523363 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.523632 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.523488 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baea2599-5723-4c5e-b063-e960ec09e121-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.523738 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.523719 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.523821 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.523795 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.523874 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.523825 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smhgj\" (UniqueName: \"kubernetes.io/projected/baea2599-5723-4c5e-b063-e960ec09e121-kube-api-access-smhgj\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.523874 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.523795 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.523996 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.523879 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.524051 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.524006 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baea2599-5723-4c5e-b063-e960ec09e121-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.524051 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.524006 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.524051 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.524027 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baea2599-5723-4c5e-b063-e960ec09e121-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.524242 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.524218 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.524579 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.524560 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baea2599-5723-4c5e-b063-e960ec09e121-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.526034 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.526007 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.527191 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.527170 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baea2599-5723-4c5e-b063-e960ec09e121-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.531974 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.531952 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baea2599-5723-4c5e-b063-e960ec09e121-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.532814 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.532789 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smhgj\" (UniqueName: \"kubernetes.io/projected/baea2599-5723-4c5e-b063-e960ec09e121-kube-api-access-smhgj\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.577910 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.577868 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:12.611140 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.611102 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:12.699397 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.695509 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:12.729786 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.727819 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" Oct 31 03:55:12.885365 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.885338 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf"] Oct 31 03:55:12.889616 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.889510 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:55:12.889616 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.889602 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:55:12.890007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.889651 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:55:12.994520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.994491 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45"] Oct 31 03:55:12.996241 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:55:12.996215 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaea2599_5723_4c5e_b063_e960ec09e121.slice/crio-fda7b4800aa5ce3a3dec477b2dfad9b1dff5c5df99ed920772d3f63ec335cada WatchSource:0}: Error finding container fda7b4800aa5ce3a3dec477b2dfad9b1dff5c5df99ed920772d3f63ec335cada: Status 404 returned error can't find the container with id fda7b4800aa5ce3a3dec477b2dfad9b1dff5c5df99ed920772d3f63ec335cada Oct 31 03:55:12.999641 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.999611 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:55:12.999748 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.999698 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:55:12.999748 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:12.999735 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:55:13.622613 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:13.622556 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:13.790671 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:13.790629 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" event={"ID":"baea2599-5723-4c5e-b063-e960ec09e121","Type":"ContainerStarted","Data":"23ad0c13c66bb7005458fde49c3ab6eb81c1bd5c866c0a93f6e33b4dd94cadda"} Oct 31 03:55:13.790891 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:13.790737 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" event={"ID":"baea2599-5723-4c5e-b063-e960ec09e121","Type":"ContainerStarted","Data":"fda7b4800aa5ce3a3dec477b2dfad9b1dff5c5df99ed920772d3f63ec335cada"} Oct 31 03:55:13.793897 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:13.793845 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" event={"ID":"01e9fcdd-b435-47dc-a064-02ff1d074156","Type":"ContainerStarted","Data":"634cf9cedaca6b5617d83f48568ecf61f6d274080ab6f25114cbe918b4fb6ca8"} Oct 31 03:55:13.794049 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:13.793916 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" event={"ID":"01e9fcdd-b435-47dc-a064-02ff1d074156","Type":"ContainerStarted","Data":"e1488f3cfaaa14b6baec0bd90795cdcfa306573e7cc97c9105e31e58162cb94e"} Oct 31 03:55:13.814736 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:13.814664 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" podStartSLOduration=1.814648839 podStartE2EDuration="1.814648839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:13.809871964 +0000 UTC m=+2348.344566919" watchObservedRunningTime="2026-10-31 03:55:13.814648839 +0000 UTC m=+2348.349344030" Oct 31 03:55:13.880830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:13.879778 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" podStartSLOduration=1.8797539680000002 podStartE2EDuration="1.879753968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:13.854088342 +0000 UTC m=+2348.388783304" watchObservedRunningTime="2026-10-31 03:55:13.879753968 +0000 UTC m=+2348.414448925" Oct 31 03:55:14.444701 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:14.440831 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:14.591721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:14.591530 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:14.593951 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:14.593921 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 31 03:55:14.594109 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:14.593979 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" podUID="01e9fcdd-b435-47dc-a064-02ff1d074156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 31 03:55:14.611950 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:14.611913 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:14.613319 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:14.613284 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 31 03:55:14.613432 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:14.613329 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" podUID="baea2599-5723-4c5e-b063-e960ec09e121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 31 03:55:15.598963 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:15.598926 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 31 03:55:15.599559 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:15.598990 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" podUID="01e9fcdd-b435-47dc-a064-02ff1d074156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 31 03:55:15.611460 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:15.611426 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 31 03:55:15.612232 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:15.611501 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" podUID="baea2599-5723-4c5e-b063-e960ec09e121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 31 03:55:16.165749 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:16.160541 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:16.353181 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:16.349429 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:16.592658 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:16.592153 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 31 03:55:16.592658 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:16.592218 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" podUID="01e9fcdd-b435-47dc-a064-02ff1d074156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 31 03:55:16.611466 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:16.611409 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 31 03:55:16.611995 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:16.611535 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" podUID="baea2599-5723-4c5e-b063-e960ec09e121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 31 03:55:17.195013 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:55:17.194962 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:55:17.421393 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:17.417411 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:17.608345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:17.607965 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 31 03:55:17.608345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:17.608024 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" podUID="01e9fcdd-b435-47dc-a064-02ff1d074156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 31 03:55:17.615805 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:17.615773 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 31 03:55:17.616342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:17.616316 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" podUID="baea2599-5723-4c5e-b063-e960ec09e121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 31 03:55:17.715631 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:17.700351 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:17.899187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:17.899091 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:18.303607 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:18.302405 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:18.483540 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:18.483487 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:18.598019 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:18.597929 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 31 03:55:18.598709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:18.598650 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" podUID="01e9fcdd-b435-47dc-a064-02ff1d074156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 31 03:55:18.622993 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:18.622943 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 31 03:55:18.623594 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:18.623557 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" podUID="baea2599-5723-4c5e-b063-e960ec09e121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 31 03:55:19.098601 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:19.098566 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:19.282793 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:19.282738 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:19.575867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:19.575661 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:19.581274 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:19.581237 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Oct 31 03:55:19.581545 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:19.581514 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" podUID="01e9fcdd-b435-47dc-a064-02ff1d074156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Oct 31 03:55:19.615073 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:19.615036 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 31 03:55:19.615470 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:19.615440 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" podUID="baea2599-5723-4c5e-b063-e960ec09e121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 31 03:55:20.420821 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:20.379900 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:20.987500 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:20.987460 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:21.332429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:21.332340 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:21.333723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:21.333599 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:21.336262 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:21.336069 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:55:21.518376 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:21.518342 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:21.519096 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:21.519072 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:21.529106 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:21.529079 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:55:21.806949 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:21.806906 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2"] Oct 31 03:55:21.807263 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:21.807219 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" podUID="cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" containerName="istio-proxy" containerID="cri-o://3d9e5f438cd0c3ceb84a8f79ab1e58deb2a516f537763689557990c38223044b" gracePeriod=30 Oct 31 03:55:21.982277 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:21.982224 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:22.786756 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:22.786719 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:23.268839 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.268794 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4"] Oct 31 03:55:23.269162 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.269131 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" podUID="b7c0d839-b3a4-463b-a3a8-3733609d4b89" containerName="maas-api" containerID="cri-o://4806f5c5bfc3514bb6e3f351df9f6f8c5248b8f144eb7263ac29a806d51a3ed2" gracePeriod=30 Oct 31 03:55:23.349717 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.349670 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:23.548493 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.547194 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:23.664493 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.664458 5490 generic.go:356] "Generic (PLEG): container finished" podID="b7c0d839-b3a4-463b-a3a8-3733609d4b89" containerID="4806f5c5bfc3514bb6e3f351df9f6f8c5248b8f144eb7263ac29a806d51a3ed2" exitCode=0 Oct 31 03:55:23.670011 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.669960 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" event={"ID":"b7c0d839-b3a4-463b-a3a8-3733609d4b89","Type":"ContainerDied","Data":"4806f5c5bfc3514bb6e3f351df9f6f8c5248b8f144eb7263ac29a806d51a3ed2"} Oct 31 03:55:23.731965 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.729460 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:23.768810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.768783 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" Oct 31 03:55:23.839941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.839842 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7c0d839-b3a4-463b-a3a8-3733609d4b89-kube-api-access-h42jp\" (UniqueName: \"kubernetes.io/projected/b7c0d839-b3a4-463b-a3a8-3733609d4b89-kube-api-access-h42jp\") pod \"b7c0d839-b3a4-463b-a3a8-3733609d4b89\" (UID: \"b7c0d839-b3a4-463b-a3a8-3733609d4b89\") " Oct 31 03:55:23.839941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.839918 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7c0d839-b3a4-463b-a3a8-3733609d4b89-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c0d839-b3a4-463b-a3a8-3733609d4b89-metrics-certs\") pod \"b7c0d839-b3a4-463b-a3a8-3733609d4b89\" (UID: \"b7c0d839-b3a4-463b-a3a8-3733609d4b89\") " Oct 31 03:55:23.840557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.839955 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7c0d839-b3a4-463b-a3a8-3733609d4b89-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c0d839-b3a4-463b-a3a8-3733609d4b89-openshift-service-ca\") pod \"b7c0d839-b3a4-463b-a3a8-3733609d4b89\" (UID: \"b7c0d839-b3a4-463b-a3a8-3733609d4b89\") " Oct 31 03:55:23.840557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.840132 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7c0d839-b3a4-463b-a3a8-3733609d4b89-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7c0d839-b3a4-463b-a3a8-3733609d4b89-maas-api-tls\") pod \"b7c0d839-b3a4-463b-a3a8-3733609d4b89\" (UID: \"b7c0d839-b3a4-463b-a3a8-3733609d4b89\") " Oct 31 03:55:23.843292 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.843258 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c0d839-b3a4-463b-a3a8-3733609d4b89-openshift-service-ca" pod "b7c0d839-b3a4-463b-a3a8-3733609d4b89" (UID: "b7c0d839-b3a4-463b-a3a8-3733609d4b89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:55:23.845113 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.845082 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c0d839-b3a4-463b-a3a8-3733609d4b89-kube-api-access-h42jp" pod "b7c0d839-b3a4-463b-a3a8-3733609d4b89" (UID: "b7c0d839-b3a4-463b-a3a8-3733609d4b89"). InnerVolumeSpecName "kube-api-access-h42jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:23.845274 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.845246 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c0d839-b3a4-463b-a3a8-3733609d4b89-metrics-certs" pod "b7c0d839-b3a4-463b-a3a8-3733609d4b89" (UID: "b7c0d839-b3a4-463b-a3a8-3733609d4b89"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:23.846799 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.846771 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c0d839-b3a4-463b-a3a8-3733609d4b89-maas-api-tls" pod "b7c0d839-b3a4-463b-a3a8-3733609d4b89" (UID: "b7c0d839-b3a4-463b-a3a8-3733609d4b89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:23.860612 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.860581 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:23.944089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.944051 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h42jp\" (UniqueName: \"kubernetes.io/projected/b7c0d839-b3a4-463b-a3a8-3733609d4b89-kube-api-access-h42jp\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:23.944089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.944091 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c0d839-b3a4-463b-a3a8-3733609d4b89-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:23.944331 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.944109 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c0d839-b3a4-463b-a3a8-3733609d4b89-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:23.944331 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:23.944123 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7c0d839-b3a4-463b-a3a8-3733609d4b89-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:24.097555 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:24.097465 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:24.650120 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:24.650081 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:24.676104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:24.676075 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" Oct 31 03:55:24.676268 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:24.676100 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4" event={"ID":"b7c0d839-b3a4-463b-a3a8-3733609d4b89","Type":"ContainerDied","Data":"24ab5e0e007b254608cdc62961f270028130d45b567c169816821b3d05fe6554"} Oct 31 03:55:24.676268 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:24.676167 5490 scope.go:122] "RemoveContainer" containerID="4806f5c5bfc3514bb6e3f351df9f6f8c5248b8f144eb7263ac29a806d51a3ed2" Oct 31 03:55:24.716793 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:24.716761 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4"] Oct 31 03:55:24.723222 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:24.723180 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-b8fd22-ddd9f864d-c6cl4"] Oct 31 03:55:24.876714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:24.873232 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:25.057789 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:25.056900 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:25.202484 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:25.202423 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:25.356494 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:25.356389 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:25.593840 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:25.593799 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7c0d839-b3a4-463b-a3a8-3733609d4b89" path="/var/lib/kubelet/pods/b7c0d839-b3a4-463b-a3a8-3733609d4b89/volumes" Oct 31 03:55:25.635366 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:25.635284 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:25.779402 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:25.779360 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:26.165630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:26.165594 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:26.550358 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:26.550262 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:26.775698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:26.775319 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:26.991579 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:26.991543 5490 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 03:55:26.991791 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:26.991612 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" podUID="cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 03:55:27.093759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.085573 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:27.146898 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.146865 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:55:27.171984 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.171661 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-envoy\") pod \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " Oct 31 03:55:27.171984 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.171874 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-data\") pod \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " Oct 31 03:55:27.171984 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.171970 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-workload-socket\") pod \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " Oct 31 03:55:27.173180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.172069 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-kube-api-access-l89tp\" (UniqueName: \"kubernetes.io/projected/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-kube-api-access-l89tp\") pod \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " Oct 31 03:55:27.173180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.172108 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istiod-ca-cert\") pod \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " Oct 31 03:55:27.173180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.172222 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-token\" (UniqueName: \"kubernetes.io/projected/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-token\") pod \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " Oct 31 03:55:27.173180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.172288 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-credential-socket\") pod \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " Oct 31 03:55:27.173180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.172349 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-workload-certs\") pod \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " Oct 31 03:55:27.173180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.172417 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-workload-socket" pod "cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" (UID: "cb3630f1-9fe3-4a38-8095-2dba83b4e1a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:27.173180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.172431 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-data" pod "cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" (UID: "cb3630f1-9fe3-4a38-8095-2dba83b4e1a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:27.173180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.172424 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-podinfo\") pod \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\" (UID: \"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7\") " Oct 31 03:55:27.173180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.172621 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istiod-ca-cert" pod "cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" (UID: "cb3630f1-9fe3-4a38-8095-2dba83b4e1a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:55:27.173180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.172985 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-workload-certs" pod "cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" (UID: "cb3630f1-9fe3-4a38-8095-2dba83b4e1a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:27.173180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.173026 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-credential-socket" pod "cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" (UID: "cb3630f1-9fe3-4a38-8095-2dba83b4e1a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:27.173695 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.173333 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:27.173695 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.173356 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:27.173695 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.173384 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:27.173695 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.173398 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:27.173695 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.173413 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:27.176428 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.176111 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-podinfo" pod "cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" (UID: "cb3630f1-9fe3-4a38-8095-2dba83b4e1a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:55:27.177203 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.177182 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-token" pod "cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" (UID: "cb3630f1-9fe3-4a38-8095-2dba83b4e1a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:27.179200 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.178997 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-envoy" pod "cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" (UID: "cb3630f1-9fe3-4a38-8095-2dba83b4e1a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:27.179200 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.179032 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-kube-api-access-l89tp" pod "cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" (UID: "cb3630f1-9fe3-4a38-8095-2dba83b4e1a7"). InnerVolumeSpecName "kube-api-access-l89tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:27.274821 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.274784 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:27.274821 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.274819 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:27.274821 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.274829 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l89tp\" (UniqueName: \"kubernetes.io/projected/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-kube-api-access-l89tp\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:27.275235 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.274844 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:27.712547 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.712506 5490 generic.go:356] "Generic (PLEG): container finished" podID="cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" containerID="3d9e5f438cd0c3ceb84a8f79ab1e58deb2a516f537763689557990c38223044b" exitCode=0 Oct 31 03:55:27.712796 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.712600 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" Oct 31 03:55:27.712796 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.712608 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" event={"ID":"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7","Type":"ContainerDied","Data":"3d9e5f438cd0c3ceb84a8f79ab1e58deb2a516f537763689557990c38223044b"} Oct 31 03:55:27.712796 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.712731 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2" event={"ID":"cb3630f1-9fe3-4a38-8095-2dba83b4e1a7","Type":"ContainerDied","Data":"9de1bc52a1f035a963b7bddbe24574a2cc62c28f7c1bbc613fad80e6d2623c16"} Oct 31 03:55:27.712796 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.712775 5490 scope.go:122] "RemoveContainer" containerID="3d9e5f438cd0c3ceb84a8f79ab1e58deb2a516f537763689557990c38223044b" Oct 31 03:55:27.726175 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.726152 5490 scope.go:122] "RemoveContainer" containerID="3d9e5f438cd0c3ceb84a8f79ab1e58deb2a516f537763689557990c38223044b" Oct 31 03:55:27.726554 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:55:27.726531 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d9e5f438cd0c3ceb84a8f79ab1e58deb2a516f537763689557990c38223044b\": container with ID starting with 3d9e5f438cd0c3ceb84a8f79ab1e58deb2a516f537763689557990c38223044b not found: ID does not exist" containerID="3d9e5f438cd0c3ceb84a8f79ab1e58deb2a516f537763689557990c38223044b" Oct 31 03:55:27.726728 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.726562 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9e5f438cd0c3ceb84a8f79ab1e58deb2a516f537763689557990c38223044b"} err="failed to get container status \"3d9e5f438cd0c3ceb84a8f79ab1e58deb2a516f537763689557990c38223044b\": rpc error: code = NotFound desc = could not find container \"3d9e5f438cd0c3ceb84a8f79ab1e58deb2a516f537763689557990c38223044b\": container with ID starting with 3d9e5f438cd0c3ceb84a8f79ab1e58deb2a516f537763689557990c38223044b not found: ID does not exist" Oct 31 03:55:27.737086 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.737055 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2"] Oct 31 03:55:27.742929 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:27.742896 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hftp2"] Oct 31 03:55:28.258699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:28.258254 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:29.114405 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:29.114371 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:29.308931 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:29.303485 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:29.578499 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:29.578465 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" path="/var/lib/kubelet/pods/cb3630f1-9fe3-4a38-8095-2dba83b4e1a7/volumes" Oct 31 03:55:30.110722 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:30.110661 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:30.685272 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:30.685234 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:30.923718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:30.919122 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:31.725903 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:31.725853 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:31.912705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:31.911094 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:32.053798 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:32.053699 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:32.286967 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:32.286919 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:32.770761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:32.770719 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:32.846335 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:32.846301 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm"] Oct 31 03:55:32.934262 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:32.934169 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:33.102842 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:33.102760 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:33.256698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:33.246167 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:33.392230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:33.392095 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:33.531306 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:33.531264 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:33.860551 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:33.860457 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:34.046097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:34.046058 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:34.327869 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:34.327820 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:34.471266 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:34.471213 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:34.599024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:34.598925 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:35.115702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:35.112620 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:35.797448 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:35.797401 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:36.302789 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.302751 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz"] Oct 31 03:55:36.304024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.303014 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" podUID="604d3706-e3e3-4fe2-a9bf-5472a41ee2f0" containerName="maas-api" containerID="cri-o://ddbdebadd3ab577b885104a38dcbb9fd9a3a9c5ae4d69ef6f0f19ef3883821a7" gracePeriod=30 Oct 31 03:55:36.390025 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.389988 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:36.522784 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.522743 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:36.678700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.678303 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:36.738940 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.738349 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" Oct 31 03:55:36.794342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.789220 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-kube-api-access-z7qrt\" (UniqueName: \"kubernetes.io/projected/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-kube-api-access-z7qrt\") pod \"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0\" (UID: \"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0\") " Oct 31 03:55:36.794342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.789336 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-openshift-service-ca\") pod \"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0\" (UID: \"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0\") " Oct 31 03:55:36.794342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.789514 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-metrics-certs\") pod \"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0\" (UID: \"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0\") " Oct 31 03:55:36.794342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.789586 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-maas-api-tls\") pod \"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0\" (UID: \"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0\") " Oct 31 03:55:36.794342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.790162 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-openshift-service-ca" pod "604d3706-e3e3-4fe2-a9bf-5472a41ee2f0" (UID: "604d3706-e3e3-4fe2-a9bf-5472a41ee2f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:55:36.794342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.790505 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:36.799401 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.799022 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-metrics-certs" pod "604d3706-e3e3-4fe2-a9bf-5472a41ee2f0" (UID: "604d3706-e3e3-4fe2-a9bf-5472a41ee2f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:36.806057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.806000 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-kube-api-access-z7qrt" pod "604d3706-e3e3-4fe2-a9bf-5472a41ee2f0" (UID: "604d3706-e3e3-4fe2-a9bf-5472a41ee2f0"). InnerVolumeSpecName "kube-api-access-z7qrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:36.814699 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.811844 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-maas-api-tls" pod "604d3706-e3e3-4fe2-a9bf-5472a41ee2f0" (UID: "604d3706-e3e3-4fe2-a9bf-5472a41ee2f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:36.816423 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.816379 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:36.898151 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.898097 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7qrt\" (UniqueName: \"kubernetes.io/projected/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-kube-api-access-z7qrt\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:36.898151 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.898148 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:36.898406 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.898164 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:36.951537 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.951496 5490 generic.go:356] "Generic (PLEG): container finished" podID="604d3706-e3e3-4fe2-a9bf-5472a41ee2f0" containerID="ddbdebadd3ab577b885104a38dcbb9fd9a3a9c5ae4d69ef6f0f19ef3883821a7" exitCode=0 Oct 31 03:55:36.951790 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.951541 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" Oct 31 03:55:36.952038 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.951558 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" event={"ID":"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0","Type":"ContainerDied","Data":"ddbdebadd3ab577b885104a38dcbb9fd9a3a9c5ae4d69ef6f0f19ef3883821a7"} Oct 31 03:55:36.952103 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.952083 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" event={"ID":"604d3706-e3e3-4fe2-a9bf-5472a41ee2f0","Type":"ContainerDied","Data":"5b6efb82677df3141f5bf8e4caab2b1402c52d4d39217a4e5bd118be49280cf3"} Oct 31 03:55:36.952151 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.952123 5490 scope.go:122] "RemoveContainer" containerID="ddbdebadd3ab577b885104a38dcbb9fd9a3a9c5ae4d69ef6f0f19ef3883821a7" Oct 31 03:55:36.965835 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.965809 5490 scope.go:122] "RemoveContainer" containerID="ddbdebadd3ab577b885104a38dcbb9fd9a3a9c5ae4d69ef6f0f19ef3883821a7" Oct 31 03:55:36.966192 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:55:36.966169 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddbdebadd3ab577b885104a38dcbb9fd9a3a9c5ae4d69ef6f0f19ef3883821a7\": container with ID starting with ddbdebadd3ab577b885104a38dcbb9fd9a3a9c5ae4d69ef6f0f19ef3883821a7 not found: ID does not exist" containerID="ddbdebadd3ab577b885104a38dcbb9fd9a3a9c5ae4d69ef6f0f19ef3883821a7" Oct 31 03:55:36.966320 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.966202 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddbdebadd3ab577b885104a38dcbb9fd9a3a9c5ae4d69ef6f0f19ef3883821a7"} err="failed to get container status \"ddbdebadd3ab577b885104a38dcbb9fd9a3a9c5ae4d69ef6f0f19ef3883821a7\": rpc error: code = NotFound desc = could not find container \"ddbdebadd3ab577b885104a38dcbb9fd9a3a9c5ae4d69ef6f0f19ef3883821a7\": container with ID starting with ddbdebadd3ab577b885104a38dcbb9fd9a3a9c5ae4d69ef6f0f19ef3883821a7 not found: ID does not exist" Oct 31 03:55:36.978145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.978118 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz"] Oct 31 03:55:36.982979 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:36.982953 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz"] Oct 31 03:55:37.115819 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:37.115603 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:37.477158 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:37.477106 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:37.590324 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:37.590288 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="604d3706-e3e3-4fe2-a9bf-5472a41ee2f0" path="/var/lib/kubelet/pods/604d3706-e3e3-4fe2-a9bf-5472a41ee2f0/volumes" Oct 31 03:55:37.613572 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:37.613412 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:37.786823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:37.782033 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:38.135449 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:38.135345 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:38.285218 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:38.285179 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:39.013598 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:39.013561 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:39.380724 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:39.380576 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:39.554763 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:39.554701 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-afb8be-769ffd8c55-hpttz" podUID="604d3706-e3e3-4fe2-a9bf-5472a41ee2f0" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.107:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:39.678725 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:39.671047 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:39.802104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:39.802058 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:40.145905 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:40.145806 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:40.898726 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:40.898643 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:41.628702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:41.625163 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:42.122241 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:42.122205 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:42.288703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:42.286936 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:43.158371 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:43.158334 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:44.046840 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:44.044211 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:44.493349 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:44.493311 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss"] Oct 31 03:55:44.493803 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:44.493577 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" podUID="da913fbf-0ba8-4607-be4c-5197a22d1a7d" containerName="maas-api" containerID="cri-o://ab9172a4dc9016f8dba10b957efc70df500d6f2605c27e6dd2ff7a99262b480a" gracePeriod=30 Oct 31 03:55:44.589909 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:44.589862 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:44.737950 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:44.737910 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:44.908711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:44.906848 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" Oct 31 03:55:44.915284 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:44.915242 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:45.031013 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.030978 5490 generic.go:356] "Generic (PLEG): container finished" podID="da913fbf-0ba8-4607-be4c-5197a22d1a7d" containerID="ab9172a4dc9016f8dba10b957efc70df500d6f2605c27e6dd2ff7a99262b480a" exitCode=0 Oct 31 03:55:45.031449 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.031020 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" Oct 31 03:55:45.032059 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.031039 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" event={"ID":"da913fbf-0ba8-4607-be4c-5197a22d1a7d","Type":"ContainerDied","Data":"ab9172a4dc9016f8dba10b957efc70df500d6f2605c27e6dd2ff7a99262b480a"} Oct 31 03:55:45.032310 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.032297 5490 scope.go:122] "RemoveContainer" containerID="ab9172a4dc9016f8dba10b957efc70df500d6f2605c27e6dd2ff7a99262b480a" Oct 31 03:55:45.032945 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.032912 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss" event={"ID":"da913fbf-0ba8-4607-be4c-5197a22d1a7d","Type":"ContainerDied","Data":"a425344e776d6d050265787d2d612ae815783a66b0b835bf358210c0ac71965b"} Oct 31 03:55:45.058578 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.058555 5490 scope.go:122] "RemoveContainer" containerID="ab9172a4dc9016f8dba10b957efc70df500d6f2605c27e6dd2ff7a99262b480a" Oct 31 03:55:45.059156 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:55:45.059114 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab9172a4dc9016f8dba10b957efc70df500d6f2605c27e6dd2ff7a99262b480a\": container with ID starting with ab9172a4dc9016f8dba10b957efc70df500d6f2605c27e6dd2ff7a99262b480a not found: ID does not exist" containerID="ab9172a4dc9016f8dba10b957efc70df500d6f2605c27e6dd2ff7a99262b480a" Oct 31 03:55:45.059281 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.059168 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab9172a4dc9016f8dba10b957efc70df500d6f2605c27e6dd2ff7a99262b480a"} err="failed to get container status \"ab9172a4dc9016f8dba10b957efc70df500d6f2605c27e6dd2ff7a99262b480a\": rpc error: code = NotFound desc = could not find container \"ab9172a4dc9016f8dba10b957efc70df500d6f2605c27e6dd2ff7a99262b480a\": container with ID starting with ab9172a4dc9016f8dba10b957efc70df500d6f2605c27e6dd2ff7a99262b480a not found: ID does not exist" Oct 31 03:55:45.093412 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.093371 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da913fbf-0ba8-4607-be4c-5197a22d1a7d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da913fbf-0ba8-4607-be4c-5197a22d1a7d-openshift-service-ca\") pod \"da913fbf-0ba8-4607-be4c-5197a22d1a7d\" (UID: \"da913fbf-0ba8-4607-be4c-5197a22d1a7d\") " Oct 31 03:55:45.093636 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.093595 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da913fbf-0ba8-4607-be4c-5197a22d1a7d-kube-api-access-hgl7j\" (UniqueName: \"kubernetes.io/projected/da913fbf-0ba8-4607-be4c-5197a22d1a7d-kube-api-access-hgl7j\") pod \"da913fbf-0ba8-4607-be4c-5197a22d1a7d\" (UID: \"da913fbf-0ba8-4607-be4c-5197a22d1a7d\") " Oct 31 03:55:45.093784 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.093646 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da913fbf-0ba8-4607-be4c-5197a22d1a7d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da913fbf-0ba8-4607-be4c-5197a22d1a7d-metrics-certs\") pod \"da913fbf-0ba8-4607-be4c-5197a22d1a7d\" (UID: \"da913fbf-0ba8-4607-be4c-5197a22d1a7d\") " Oct 31 03:55:45.093871 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.093842 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da913fbf-0ba8-4607-be4c-5197a22d1a7d-openshift-service-ca" pod "da913fbf-0ba8-4607-be4c-5197a22d1a7d" (UID: "da913fbf-0ba8-4607-be4c-5197a22d1a7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:55:45.093965 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.093944 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da913fbf-0ba8-4607-be4c-5197a22d1a7d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da913fbf-0ba8-4607-be4c-5197a22d1a7d-maas-api-tls\") pod \"da913fbf-0ba8-4607-be4c-5197a22d1a7d\" (UID: \"da913fbf-0ba8-4607-be4c-5197a22d1a7d\") " Oct 31 03:55:45.094586 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.094553 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da913fbf-0ba8-4607-be4c-5197a22d1a7d-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:45.096199 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.096166 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da913fbf-0ba8-4607-be4c-5197a22d1a7d-maas-api-tls" pod "da913fbf-0ba8-4607-be4c-5197a22d1a7d" (UID: "da913fbf-0ba8-4607-be4c-5197a22d1a7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:45.096380 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.096357 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da913fbf-0ba8-4607-be4c-5197a22d1a7d-kube-api-access-hgl7j" pod "da913fbf-0ba8-4607-be4c-5197a22d1a7d" (UID: "da913fbf-0ba8-4607-be4c-5197a22d1a7d"). InnerVolumeSpecName "kube-api-access-hgl7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:45.096504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.096485 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da913fbf-0ba8-4607-be4c-5197a22d1a7d-metrics-certs" pod "da913fbf-0ba8-4607-be4c-5197a22d1a7d" (UID: "da913fbf-0ba8-4607-be4c-5197a22d1a7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:45.187387 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.187345 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:45.196012 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.195979 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da913fbf-0ba8-4607-be4c-5197a22d1a7d-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:45.196012 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.196009 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgl7j\" (UniqueName: \"kubernetes.io/projected/da913fbf-0ba8-4607-be4c-5197a22d1a7d-kube-api-access-hgl7j\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:45.196229 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.196023 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da913fbf-0ba8-4607-be4c-5197a22d1a7d-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:45.336186 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.335966 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:45.367742 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.367701 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss"] Oct 31 03:55:45.373384 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.373353 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b81d65-5d7465dddb-jqtss"] Oct 31 03:55:45.579639 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.579601 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da913fbf-0ba8-4607-be4c-5197a22d1a7d" path="/var/lib/kubelet/pods/da913fbf-0ba8-4607-be4c-5197a22d1a7d/volumes" Oct 31 03:55:45.870700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:45.861189 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:46.626803 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:46.626752 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:47.510760 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:55:47.510044 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:55:48.006700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:47.992436 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" containerID="cri-o://a327decf5510d4892a78d5c8b9e4449857b39a1d25950f1a99df9329619c4c38" gracePeriod=45 Oct 31 03:55:48.205703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:48.205335 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:48.392757 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:48.392387 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:48.759588 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:48.737993 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:48.819722 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:48.790086 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/ready\": dial tcp 10.133.0.128:8000: connect: connection refused" Oct 31 03:55:48.917711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:48.917592 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:49.183743 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:49.101415 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:49.516495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:49.367976 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:49.516495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:49.381939 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:49.516495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:49.491284 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:49.516495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:49.491574 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:49.516495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:49.492897 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 31 03:55:49.516495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:49.494236 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 31 03:55:49.516495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:49.495291 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 03:55:49.516495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:49.514761 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded" Oct 31 03:55:49.546643 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:49.546604 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8"] Oct 31 03:55:49.547163 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:49.547127 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" podUID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" containerName="istio-proxy" containerID="cri-o://17c7ea0c1b9baf2b2884295852991be2d2d02e408dbf636ae882a55320ae0c09" gracePeriod=30 Oct 31 03:55:49.726703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:49.617885 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/ready\": dial tcp 10.133.0.128:8000: connect: connection refused" Oct 31 03:55:49.861076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:49.860983 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:55:49.880818 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:49.880774 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 03:55:50.359272 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:50.359216 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:50.456952 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:50.456883 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:55:50.457170 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:50.456979 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" podUID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:55:50.520888 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:50.517415 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:50.583179 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:50.583152 5490 generic.go:356] "Generic (PLEG): container finished" podID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerID="a327decf5510d4892a78d5c8b9e4449857b39a1d25950f1a99df9329619c4c38" exitCode=0 Oct 31 03:55:50.584218 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:50.583308 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" event={"ID":"7880b1c4-cc27-4692-ae95-5682724c2f4c","Type":"ContainerDied","Data":"a327decf5510d4892a78d5c8b9e4449857b39a1d25950f1a99df9329619c4c38"} Oct 31 03:55:50.609397 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:50.609354 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.128:8000/ready\": dial tcp 10.133.0.128:8000: connect: connection refused" Oct 31 03:55:50.851111 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:50.851073 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:51.015482 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.015406 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:55:51.170742 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.170707 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7880b1c4-cc27-4692-ae95-5682724c2f4c-kube-api-access-jctxp\" (UniqueName: \"kubernetes.io/projected/7880b1c4-cc27-4692-ae95-5682724c2f4c-kube-api-access-jctxp\") pod \"7880b1c4-cc27-4692-ae95-5682724c2f4c\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " Oct 31 03:55:51.170971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.170820 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7880b1c4-cc27-4692-ae95-5682724c2f4c-tls-certs\" (UniqueName: \"kubernetes.io/secret/7880b1c4-cc27-4692-ae95-5682724c2f4c-tls-certs\") pod \"7880b1c4-cc27-4692-ae95-5682724c2f4c\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " Oct 31 03:55:51.170971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.170911 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-dshm\") pod \"7880b1c4-cc27-4692-ae95-5682724c2f4c\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " Oct 31 03:55:51.171104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.170992 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-tmp-dir\") pod \"7880b1c4-cc27-4692-ae95-5682724c2f4c\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " Oct 31 03:55:51.171104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.171030 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-model-cache\") pod \"7880b1c4-cc27-4692-ae95-5682724c2f4c\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " Oct 31 03:55:51.171104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.171081 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-home\" (UniqueName: \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-home\") pod \"7880b1c4-cc27-4692-ae95-5682724c2f4c\" (UID: \"7880b1c4-cc27-4692-ae95-5682724c2f4c\") " Oct 31 03:55:51.171950 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.171915 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-tmp-dir" pod "7880b1c4-cc27-4692-ae95-5682724c2f4c" (UID: "7880b1c4-cc27-4692-ae95-5682724c2f4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:51.172120 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.172101 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-model-cache" pod "7880b1c4-cc27-4692-ae95-5682724c2f4c" (UID: "7880b1c4-cc27-4692-ae95-5682724c2f4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:51.172197 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.172151 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-home" pod "7880b1c4-cc27-4692-ae95-5682724c2f4c" (UID: "7880b1c4-cc27-4692-ae95-5682724c2f4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:51.173655 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.173622 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-dshm" pod "7880b1c4-cc27-4692-ae95-5682724c2f4c" (UID: "7880b1c4-cc27-4692-ae95-5682724c2f4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:51.173968 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.173947 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7880b1c4-cc27-4692-ae95-5682724c2f4c-tls-certs" pod "7880b1c4-cc27-4692-ae95-5682724c2f4c" (UID: "7880b1c4-cc27-4692-ae95-5682724c2f4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:55:51.175396 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.175359 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7880b1c4-cc27-4692-ae95-5682724c2f4c-kube-api-access-jctxp" pod "7880b1c4-cc27-4692-ae95-5682724c2f4c" (UID: "7880b1c4-cc27-4692-ae95-5682724c2f4c"). InnerVolumeSpecName "kube-api-access-jctxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:51.273867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.273764 5490 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7880b1c4-cc27-4692-ae95-5682724c2f4c-tls-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:51.273867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.273800 5490 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-dshm\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:51.273867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.273836 5490 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-tmp-dir\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:51.273867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.273848 5490 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-model-cache\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:51.273867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.273862 5490 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7880b1c4-cc27-4692-ae95-5682724c2f4c-home\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:51.274295 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.273878 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jctxp\" (UniqueName: \"kubernetes.io/projected/7880b1c4-cc27-4692-ae95-5682724c2f4c-kube-api-access-jctxp\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:51.414768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.404794 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:51.593976 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.593951 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" Oct 31 03:55:51.593976 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.593960 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm" event={"ID":"7880b1c4-cc27-4692-ae95-5682724c2f4c","Type":"ContainerDied","Data":"608cd445bdaea1b94221fd9c1a3b62916c4ec5f5541be75ca03412fb80187a55"} Oct 31 03:55:51.594187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.594014 5490 scope.go:122] "RemoveContainer" containerID="a327decf5510d4892a78d5c8b9e4449857b39a1d25950f1a99df9329619c4c38" Oct 31 03:55:51.614364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.614328 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm"] Oct 31 03:55:51.618880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:51.618853 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-50b5f2/test-model-50b5f2-kserve-978bd677f-2clcm"] Oct 31 03:55:52.157702 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:52.157245 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:53.025540 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:53.025499 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:53.577193 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:53.577161 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" path="/var/lib/kubelet/pods/7880b1c4-cc27-4692-ae95-5682724c2f4c/volumes" Oct 31 03:55:53.778731 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:53.778696 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:54.442178 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:54.442142 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:55.149969 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.145771 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:55.271823 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.271793 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:55:55.357188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.357149 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-envoy\") pod \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " Oct 31 03:55:55.357386 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.357202 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-credential-socket\") pod \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " Oct 31 03:55:55.357386 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.357227 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-workload-socket\") pod \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " Oct 31 03:55:55.357386 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.357281 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-podinfo\") pod \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " Oct 31 03:55:55.357386 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.357310 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-workload-certs\") pod \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " Oct 31 03:55:55.357628 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.357399 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-data\") pod \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " Oct 31 03:55:55.357628 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.357439 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-kube-api-access-85mjd\" (UniqueName: \"kubernetes.io/projected/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-kube-api-access-85mjd\") pod \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " Oct 31 03:55:55.357628 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.357471 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-token\" (UniqueName: \"kubernetes.io/projected/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-token\") pod \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " Oct 31 03:55:55.357866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.357649 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-workload-socket" pod "f91c38ef-6298-4315-9a8f-bedd67fcdd1c" (UID: "f91c38ef-6298-4315-9a8f-bedd67fcdd1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:55.357866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.357788 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istiod-ca-cert\") pod \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\" (UID: \"f91c38ef-6298-4315-9a8f-bedd67fcdd1c\") " Oct 31 03:55:55.358646 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.358614 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-credential-socket" pod "f91c38ef-6298-4315-9a8f-bedd67fcdd1c" (UID: "f91c38ef-6298-4315-9a8f-bedd67fcdd1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:55.358646 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.358631 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-workload-certs" pod "f91c38ef-6298-4315-9a8f-bedd67fcdd1c" (UID: "f91c38ef-6298-4315-9a8f-bedd67fcdd1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:55.359007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.358971 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:55.359007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.359003 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:55.359188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.359020 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:55.359600 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.359568 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-data" pod "f91c38ef-6298-4315-9a8f-bedd67fcdd1c" (UID: "f91c38ef-6298-4315-9a8f-bedd67fcdd1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:55.359788 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.359767 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istiod-ca-cert" pod "f91c38ef-6298-4315-9a8f-bedd67fcdd1c" (UID: "f91c38ef-6298-4315-9a8f-bedd67fcdd1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:55:55.360543 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.360522 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-podinfo" pod "f91c38ef-6298-4315-9a8f-bedd67fcdd1c" (UID: "f91c38ef-6298-4315-9a8f-bedd67fcdd1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:55:55.360786 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.360753 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-envoy" pod "f91c38ef-6298-4315-9a8f-bedd67fcdd1c" (UID: "f91c38ef-6298-4315-9a8f-bedd67fcdd1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:55:55.360901 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.360875 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-token" pod "f91c38ef-6298-4315-9a8f-bedd67fcdd1c" (UID: "f91c38ef-6298-4315-9a8f-bedd67fcdd1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:55.361282 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.361261 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-kube-api-access-85mjd" pod "f91c38ef-6298-4315-9a8f-bedd67fcdd1c" (UID: "f91c38ef-6298-4315-9a8f-bedd67fcdd1c"). InnerVolumeSpecName "kube-api-access-85mjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:55:55.460520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.460488 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:55.460520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.460517 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:55.460520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.460527 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:55.461145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.460536 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:55.461145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.460545 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85mjd\" (UniqueName: \"kubernetes.io/projected/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-kube-api-access-85mjd\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:55.461145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.460556 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f91c38ef-6298-4315-9a8f-bedd67fcdd1c-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:55:55.627277 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.627242 5490 generic.go:356] "Generic (PLEG): container finished" podID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" containerID="17c7ea0c1b9baf2b2884295852991be2d2d02e408dbf636ae882a55320ae0c09" exitCode=0 Oct 31 03:55:55.627463 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.627304 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" Oct 31 03:55:55.627463 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.627341 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" event={"ID":"f91c38ef-6298-4315-9a8f-bedd67fcdd1c","Type":"ContainerDied","Data":"17c7ea0c1b9baf2b2884295852991be2d2d02e408dbf636ae882a55320ae0c09"} Oct 31 03:55:55.627463 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.627418 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8" event={"ID":"f91c38ef-6298-4315-9a8f-bedd67fcdd1c","Type":"ContainerDied","Data":"c3e01c3c583bca436c46e3783b30492a4af398611d0f66b369de6584db1d68b6"} Oct 31 03:55:55.627463 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.627449 5490 scope.go:122] "RemoveContainer" containerID="17c7ea0c1b9baf2b2884295852991be2d2d02e408dbf636ae882a55320ae0c09" Oct 31 03:55:55.642406 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.642378 5490 scope.go:122] "RemoveContainer" containerID="13b21d79d292a339aad82c96083618172b35ee0058c235ccc96c8e15faa8a955" Oct 31 03:55:55.651016 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.650989 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8"] Oct 31 03:55:55.655504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.655479 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-djfp8"] Oct 31 03:55:55.657797 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.657776 5490 scope.go:122] "RemoveContainer" containerID="17c7ea0c1b9baf2b2884295852991be2d2d02e408dbf636ae882a55320ae0c09" Oct 31 03:55:55.658143 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:55:55.658113 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17c7ea0c1b9baf2b2884295852991be2d2d02e408dbf636ae882a55320ae0c09\": container with ID starting with 17c7ea0c1b9baf2b2884295852991be2d2d02e408dbf636ae882a55320ae0c09 not found: ID does not exist" containerID="17c7ea0c1b9baf2b2884295852991be2d2d02e408dbf636ae882a55320ae0c09" Oct 31 03:55:55.658233 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.658149 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17c7ea0c1b9baf2b2884295852991be2d2d02e408dbf636ae882a55320ae0c09"} err="failed to get container status \"17c7ea0c1b9baf2b2884295852991be2d2d02e408dbf636ae882a55320ae0c09\": rpc error: code = NotFound desc = could not find container \"17c7ea0c1b9baf2b2884295852991be2d2d02e408dbf636ae882a55320ae0c09\": container with ID starting with 17c7ea0c1b9baf2b2884295852991be2d2d02e408dbf636ae882a55320ae0c09 not found: ID does not exist" Oct 31 03:55:55.658233 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.658168 5490 scope.go:122] "RemoveContainer" containerID="13b21d79d292a339aad82c96083618172b35ee0058c235ccc96c8e15faa8a955" Oct 31 03:55:55.658453 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:55:55.658435 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b21d79d292a339aad82c96083618172b35ee0058c235ccc96c8e15faa8a955\": container with ID starting with 13b21d79d292a339aad82c96083618172b35ee0058c235ccc96c8e15faa8a955 not found: ID does not exist" containerID="13b21d79d292a339aad82c96083618172b35ee0058c235ccc96c8e15faa8a955" Oct 31 03:55:55.658509 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.658463 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b21d79d292a339aad82c96083618172b35ee0058c235ccc96c8e15faa8a955"} err="failed to get container status \"13b21d79d292a339aad82c96083618172b35ee0058c235ccc96c8e15faa8a955\": rpc error: code = NotFound desc = could not find container \"13b21d79d292a339aad82c96083618172b35ee0058c235ccc96c8e15faa8a955\": container with ID starting with 13b21d79d292a339aad82c96083618172b35ee0058c235ccc96c8e15faa8a955 not found: ID does not exist" Oct 31 03:55:55.882082 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:55.881988 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:56.022444 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:56.022409 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:56.629289 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:56.629247 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:56.985153 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:56.985076 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:57.279480 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:57.278123 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn"] Oct 31 03:55:57.279480 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:57.278361 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" podUID="015afcb4-d156-4163-a342-f84471c9dd51" containerName="istio-proxy" containerID="cri-o://b047fbd7d43f633432b3e6284e0e4cd0557b8e084c0bb6dabeca99dade16a78b" gracePeriod=30 Oct 31 03:55:57.351527 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:57.351473 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:57.579578 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:57.579504 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" path="/var/lib/kubelet/pods/f91c38ef-6298-4315-9a8f-bedd67fcdd1c/volumes" Oct 31 03:55:57.656479 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:57.656391 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:57.796468 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:57.796429 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:58.104032 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:58.103997 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:58.847410 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:58.847370 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:59.512475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:59.511981 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:59.799421 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:59.799335 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:59.935472 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:59.935436 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:55:59.986925 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:55:59.986889 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd"] Oct 31 03:56:00.096761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:00.086105 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:00.221528 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:00.221488 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:00.414854 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:00.414814 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:00.674093 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:00.674061 5490 generic.go:356] "Generic (PLEG): container finished" podID="24a03b21-4c4f-422e-baa4-6e905b1b0bf5" containerID="770d923a22550d4a90ab065ee3eb02df2ca0b1192f8fe0fbe642e55baff56e6b" exitCode=0 Oct 31 03:56:00.674317 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:00.674134 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" event={"ID":"24a03b21-4c4f-422e-baa4-6e905b1b0bf5","Type":"ContainerDied","Data":"770d923a22550d4a90ab065ee3eb02df2ca0b1192f8fe0fbe642e55baff56e6b"} Oct 31 03:56:00.674752 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:00.674732 5490 scope.go:122] "RemoveContainer" containerID="770d923a22550d4a90ab065ee3eb02df2ca0b1192f8fe0fbe642e55baff56e6b" Oct 31 03:56:00.676856 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:00.676818 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:56:00.676948 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:00.676913 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:56:00.676989 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:00.676960 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:56:01.062130 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:01.062037 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:01.490798 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:01.490764 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:56:01.685578 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:01.685542 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" event={"ID":"24a03b21-4c4f-422e-baa4-6e905b1b0bf5","Type":"ContainerStarted","Data":"ebbcf30bc2205d2f85a6a78f2f8b275f7f73bac58e4826c35d31601c7ee0a9e7"} Oct 31 03:56:01.799711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:01.797954 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:01.965793 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:01.965754 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:56:01.967448 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:01.967417 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 31 03:56:01.967580 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:01.967471 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" podUID="24a03b21-4c4f-422e-baa4-6e905b1b0bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 31 03:56:02.679838 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.679732 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:02.707563 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.707521 5490 generic.go:356] "Generic (PLEG): container finished" podID="015afcb4-d156-4163-a342-f84471c9dd51" containerID="b047fbd7d43f633432b3e6284e0e4cd0557b8e084c0bb6dabeca99dade16a78b" exitCode=0 Oct 31 03:56:02.707808 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.707569 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" event={"ID":"015afcb4-d156-4163-a342-f84471c9dd51","Type":"ContainerDied","Data":"b047fbd7d43f633432b3e6284e0e4cd0557b8e084c0bb6dabeca99dade16a78b"} Oct 31 03:56:02.707808 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.707652 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" event={"ID":"015afcb4-d156-4163-a342-f84471c9dd51","Type":"ContainerDied","Data":"4d6fe02dc3b811e0e4998d609b39c0b1a06080d1d7940afa95bd13f181e9d4a8"} Oct 31 03:56:02.707808 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.707693 5490 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d6fe02dc3b811e0e4998d609b39c0b1a06080d1d7940afa95bd13f181e9d4a8" Oct 31 03:56:02.707808 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.707713 5490 scope.go:122] "RemoveContainer" containerID="86094c517a636ad0fb95d0395d8f04eaf0b71ce63609059fa53c252f4c2da1e7" Oct 31 03:56:02.718361 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.718335 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:56:02.780896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.780864 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-istio-envoy\") pod \"015afcb4-d156-4163-a342-f84471c9dd51\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " Oct 31 03:56:02.781100 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.780923 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-credential-socket\") pod \"015afcb4-d156-4163-a342-f84471c9dd51\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " Oct 31 03:56:02.781100 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.780955 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/015afcb4-d156-4163-a342-f84471c9dd51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/015afcb4-d156-4163-a342-f84471c9dd51-istiod-ca-cert\") pod \"015afcb4-d156-4163-a342-f84471c9dd51\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " Oct 31 03:56:02.781100 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.781034 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-workload-certs\") pod \"015afcb4-d156-4163-a342-f84471c9dd51\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " Oct 31 03:56:02.781100 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.781082 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/015afcb4-d156-4163-a342-f84471c9dd51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/015afcb4-d156-4163-a342-f84471c9dd51-istio-podinfo\") pod \"015afcb4-d156-4163-a342-f84471c9dd51\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " Oct 31 03:56:02.781342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.781182 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/015afcb4-d156-4163-a342-f84471c9dd51-kube-api-access-6kr8z\" (UniqueName: \"kubernetes.io/projected/015afcb4-d156-4163-a342-f84471c9dd51-kube-api-access-6kr8z\") pod \"015afcb4-d156-4163-a342-f84471c9dd51\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " Oct 31 03:56:02.781342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.781219 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-istio-data\") pod \"015afcb4-d156-4163-a342-f84471c9dd51\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " Oct 31 03:56:02.781342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.781323 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-credential-socket" pod "015afcb4-d156-4163-a342-f84471c9dd51" (UID: "015afcb4-d156-4163-a342-f84471c9dd51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:02.781498 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.781335 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-workload-certs" pod "015afcb4-d156-4163-a342-f84471c9dd51" (UID: "015afcb4-d156-4163-a342-f84471c9dd51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:02.781498 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.781426 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/015afcb4-d156-4163-a342-f84471c9dd51-istiod-ca-cert" pod "015afcb4-d156-4163-a342-f84471c9dd51" (UID: "015afcb4-d156-4163-a342-f84471c9dd51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:02.781853 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.781827 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-istio-data" pod "015afcb4-d156-4163-a342-f84471c9dd51" (UID: "015afcb4-d156-4163-a342-f84471c9dd51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:02.782162 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.782142 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/015afcb4-d156-4163-a342-f84471c9dd51-istio-token\" (UniqueName: \"kubernetes.io/projected/015afcb4-d156-4163-a342-f84471c9dd51-istio-token\") pod \"015afcb4-d156-4163-a342-f84471c9dd51\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " Oct 31 03:56:02.782289 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.782274 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-workload-socket\") pod \"015afcb4-d156-4163-a342-f84471c9dd51\" (UID: \"015afcb4-d156-4163-a342-f84471c9dd51\") " Oct 31 03:56:02.782588 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.782565 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-workload-socket" pod "015afcb4-d156-4163-a342-f84471c9dd51" (UID: "015afcb4-d156-4163-a342-f84471c9dd51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:02.783065 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.783042 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:02.783065 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.783066 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:02.783234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.783077 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:02.783234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.783086 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:02.783234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.783095 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/015afcb4-d156-4163-a342-f84471c9dd51-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:02.783987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.783962 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/015afcb4-d156-4163-a342-f84471c9dd51-istio-podinfo" pod "015afcb4-d156-4163-a342-f84471c9dd51" (UID: "015afcb4-d156-4163-a342-f84471c9dd51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:02.784367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.784342 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-istio-envoy" pod "015afcb4-d156-4163-a342-f84471c9dd51" (UID: "015afcb4-d156-4163-a342-f84471c9dd51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:02.785089 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.785063 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015afcb4-d156-4163-a342-f84471c9dd51-kube-api-access-6kr8z" pod "015afcb4-d156-4163-a342-f84471c9dd51" (UID: "015afcb4-d156-4163-a342-f84471c9dd51"). InnerVolumeSpecName "kube-api-access-6kr8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:02.785184 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.785130 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015afcb4-d156-4163-a342-f84471c9dd51-istio-token" pod "015afcb4-d156-4163-a342-f84471c9dd51" (UID: "015afcb4-d156-4163-a342-f84471c9dd51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:02.884482 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.884441 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/015afcb4-d156-4163-a342-f84471c9dd51-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:02.884482 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.884473 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kr8z\" (UniqueName: \"kubernetes.io/projected/015afcb4-d156-4163-a342-f84471c9dd51-kube-api-access-6kr8z\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:02.884482 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.884483 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/015afcb4-d156-4163-a342-f84471c9dd51-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:02.884482 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.884491 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/015afcb4-d156-4163-a342-f84471c9dd51-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:02.966102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.966008 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 31 03:56:02.966408 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:02.966099 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" podUID="24a03b21-4c4f-422e-baa4-6e905b1b0bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 31 03:56:03.515267 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:03.515217 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:03.717859 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:03.717832 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn" Oct 31 03:56:03.769748 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:03.769607 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn"] Oct 31 03:56:03.774055 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:03.774026 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-lvzdn"] Oct 31 03:56:03.968259 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:03.968217 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 31 03:56:03.968471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:03.968303 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" podUID="24a03b21-4c4f-422e-baa4-6e905b1b0bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 31 03:56:04.206135 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:04.206087 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:04.922354 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:04.921532 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:04.966824 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:04.966785 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 31 03:56:04.967156 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:04.967123 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" podUID="24a03b21-4c4f-422e-baa4-6e905b1b0bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 31 03:56:04.988409 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:04.988323 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" podUID="bae51678-b17d-4a9c-98da-77d3ecf0d6ed" containerName="payload-pre-processing" containerID="cri-o://bbacbd77c0b5d8febb4f46075637f1864b7a1d701c8274afde83b6e0d191211f" gracePeriod=25 Oct 31 03:56:05.341296 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.337896 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" Oct 31 03:56:05.476215 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.476137 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae51678-b17d-4a9c-98da-77d3ecf0d6ed-kube-api-access-lw7jp\" (UniqueName: \"kubernetes.io/projected/bae51678-b17d-4a9c-98da-77d3ecf0d6ed-kube-api-access-lw7jp\") pod \"bae51678-b17d-4a9c-98da-77d3ecf0d6ed\" (UID: \"bae51678-b17d-4a9c-98da-77d3ecf0d6ed\") " Oct 31 03:56:05.476484 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.476446 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae51678-b17d-4a9c-98da-77d3ecf0d6ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae51678-b17d-4a9c-98da-77d3ecf0d6ed-plugins-config-volume\") pod \"bae51678-b17d-4a9c-98da-77d3ecf0d6ed\" (UID: \"bae51678-b17d-4a9c-98da-77d3ecf0d6ed\") " Oct 31 03:56:05.476819 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.476791 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae51678-b17d-4a9c-98da-77d3ecf0d6ed-plugins-config-volume" pod "bae51678-b17d-4a9c-98da-77d3ecf0d6ed" (UID: "bae51678-b17d-4a9c-98da-77d3ecf0d6ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:05.477154 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.477135 5490 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae51678-b17d-4a9c-98da-77d3ecf0d6ed-plugins-config-volume\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:05.478355 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.478336 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae51678-b17d-4a9c-98da-77d3ecf0d6ed-kube-api-access-lw7jp" pod "bae51678-b17d-4a9c-98da-77d3ecf0d6ed" (UID: "bae51678-b17d-4a9c-98da-77d3ecf0d6ed"). InnerVolumeSpecName "kube-api-access-lw7jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:05.501808 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.501781 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:05.579965 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.579933 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw7jp\" (UniqueName: \"kubernetes.io/projected/bae51678-b17d-4a9c-98da-77d3ecf0d6ed-kube-api-access-lw7jp\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:05.584041 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.584001 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="015afcb4-d156-4163-a342-f84471c9dd51" path="/var/lib/kubelet/pods/015afcb4-d156-4163-a342-f84471c9dd51/volumes" Oct 31 03:56:05.683704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.683440 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:05.800759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.800159 5490 generic.go:356] "Generic (PLEG): container finished" podID="bae51678-b17d-4a9c-98da-77d3ecf0d6ed" containerID="bbacbd77c0b5d8febb4f46075637f1864b7a1d701c8274afde83b6e0d191211f" exitCode=0 Oct 31 03:56:05.800759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.800437 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" event={"ID":"bae51678-b17d-4a9c-98da-77d3ecf0d6ed","Type":"ContainerDied","Data":"bbacbd77c0b5d8febb4f46075637f1864b7a1d701c8274afde83b6e0d191211f"} Oct 31 03:56:05.800759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.800508 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" event={"ID":"bae51678-b17d-4a9c-98da-77d3ecf0d6ed","Type":"ContainerDied","Data":"3ec677065186efa154227bf21d684683f8dae74dc91b0f47c89d27006427cad9"} Oct 31 03:56:05.800759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.800540 5490 scope.go:122] "RemoveContainer" containerID="bbacbd77c0b5d8febb4f46075637f1864b7a1d701c8274afde83b6e0d191211f" Oct 31 03:56:05.802355 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.801186 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd" Oct 31 03:56:05.820194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.819302 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:05.878737 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.873917 5490 scope.go:122] "RemoveContainer" containerID="bbacbd77c0b5d8febb4f46075637f1864b7a1d701c8274afde83b6e0d191211f" Oct 31 03:56:05.878737 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:05.874422 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbacbd77c0b5d8febb4f46075637f1864b7a1d701c8274afde83b6e0d191211f\": container with ID starting with bbacbd77c0b5d8febb4f46075637f1864b7a1d701c8274afde83b6e0d191211f not found: ID does not exist" containerID="bbacbd77c0b5d8febb4f46075637f1864b7a1d701c8274afde83b6e0d191211f" Oct 31 03:56:05.878737 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.874464 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbacbd77c0b5d8febb4f46075637f1864b7a1d701c8274afde83b6e0d191211f"} err="failed to get container status \"bbacbd77c0b5d8febb4f46075637f1864b7a1d701c8274afde83b6e0d191211f\": rpc error: code = NotFound desc = could not find container \"bbacbd77c0b5d8febb4f46075637f1864b7a1d701c8274afde83b6e0d191211f\": container with ID starting with bbacbd77c0b5d8febb4f46075637f1864b7a1d701c8274afde83b6e0d191211f not found: ID does not exist" Oct 31 03:56:05.889720 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.881721 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd"] Oct 31 03:56:05.899143 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.899078 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-86a834-5d8676bd7f-g4ndd"] Oct 31 03:56:05.965867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.965823 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Oct 31 03:56:05.966310 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:05.965916 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" podUID="24a03b21-4c4f-422e-baa4-6e905b1b0bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Oct 31 03:56:06.084356 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:06.084256 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:06.459230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:06.459150 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:06.996586 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:06.996520 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:07.015062 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:07.015030 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:56:07.017861 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:07.017830 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:56:07.029864 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:07.029830 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:56:07.579975 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:07.579938 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae51678-b17d-4a9c-98da-77d3ecf0d6ed" path="/var/lib/kubelet/pods/bae51678-b17d-4a9c-98da-77d3ecf0d6ed/volumes" Oct 31 03:56:07.924080 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:07.924040 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:08.075989 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:08.075945 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:08.239759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:08.233222 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:08.669293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:08.669254 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg"] Oct 31 03:56:08.669511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:08.669490 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" podUID="b8353165-d9b4-445a-9a11-5fdf04307860" containerName="maas-api" containerID="cri-o://588ea823d809e73e387ebb9713e32ba6c9e4aa18c019b7a332dfc166259a4490" gracePeriod=30 Oct 31 03:56:08.786194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:08.786145 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:08.833807 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:08.833772 5490 generic.go:356] "Generic (PLEG): container finished" podID="b8353165-d9b4-445a-9a11-5fdf04307860" containerID="588ea823d809e73e387ebb9713e32ba6c9e4aa18c019b7a332dfc166259a4490" exitCode=0 Oct 31 03:56:08.834030 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:08.833842 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" event={"ID":"b8353165-d9b4-445a-9a11-5fdf04307860","Type":"ContainerDied","Data":"588ea823d809e73e387ebb9713e32ba6c9e4aa18c019b7a332dfc166259a4490"} Oct 31 03:56:08.935775 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:08.931185 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:08.997187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:08.997157 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" Oct 31 03:56:09.076025 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.075961 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:09.164698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.159495 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8353165-d9b4-445a-9a11-5fdf04307860-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8353165-d9b4-445a-9a11-5fdf04307860-openshift-service-ca\") pod \"b8353165-d9b4-445a-9a11-5fdf04307860\" (UID: \"b8353165-d9b4-445a-9a11-5fdf04307860\") " Oct 31 03:56:09.164698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.159756 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8353165-d9b4-445a-9a11-5fdf04307860-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8353165-d9b4-445a-9a11-5fdf04307860-maas-api-tls\") pod \"b8353165-d9b4-445a-9a11-5fdf04307860\" (UID: \"b8353165-d9b4-445a-9a11-5fdf04307860\") " Oct 31 03:56:09.164698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.159810 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8353165-d9b4-445a-9a11-5fdf04307860-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8353165-d9b4-445a-9a11-5fdf04307860-metrics-certs\") pod \"b8353165-d9b4-445a-9a11-5fdf04307860\" (UID: \"b8353165-d9b4-445a-9a11-5fdf04307860\") " Oct 31 03:56:09.164698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.159894 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8353165-d9b4-445a-9a11-5fdf04307860-kube-api-access-nrbw7\" (UniqueName: \"kubernetes.io/projected/b8353165-d9b4-445a-9a11-5fdf04307860-kube-api-access-nrbw7\") pod \"b8353165-d9b4-445a-9a11-5fdf04307860\" (UID: \"b8353165-d9b4-445a-9a11-5fdf04307860\") " Oct 31 03:56:09.164698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.162601 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8353165-d9b4-445a-9a11-5fdf04307860-openshift-service-ca" pod "b8353165-d9b4-445a-9a11-5fdf04307860" (UID: "b8353165-d9b4-445a-9a11-5fdf04307860"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:09.205992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.205944 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8353165-d9b4-445a-9a11-5fdf04307860-maas-api-tls" pod "b8353165-d9b4-445a-9a11-5fdf04307860" (UID: "b8353165-d9b4-445a-9a11-5fdf04307860"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:09.207671 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.207619 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8353165-d9b4-445a-9a11-5fdf04307860-kube-api-access-nrbw7" pod "b8353165-d9b4-445a-9a11-5fdf04307860" (UID: "b8353165-d9b4-445a-9a11-5fdf04307860"). InnerVolumeSpecName "kube-api-access-nrbw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:09.207941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.207918 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8353165-d9b4-445a-9a11-5fdf04307860-metrics-certs" pod "b8353165-d9b4-445a-9a11-5fdf04307860" (UID: "b8353165-d9b4-445a-9a11-5fdf04307860"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:09.223384 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.223287 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:09.262723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.262512 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8353165-d9b4-445a-9a11-5fdf04307860-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:09.262723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.262555 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8353165-d9b4-445a-9a11-5fdf04307860-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:09.262723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.262569 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8353165-d9b4-445a-9a11-5fdf04307860-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:09.262723 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.262584 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrbw7\" (UniqueName: \"kubernetes.io/projected/b8353165-d9b4-445a-9a11-5fdf04307860-kube-api-access-nrbw7\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:09.563992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.563891 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:09.843872 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.843783 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" event={"ID":"b8353165-d9b4-445a-9a11-5fdf04307860","Type":"ContainerDied","Data":"e83f449fb0aa807abdfafa20285861ffc5da367660c4bcbbe1ae63e468f7744f"} Oct 31 03:56:09.843872 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.843813 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg" Oct 31 03:56:09.843872 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.843839 5490 scope.go:122] "RemoveContainer" containerID="588ea823d809e73e387ebb9713e32ba6c9e4aa18c019b7a332dfc166259a4490" Oct 31 03:56:09.894996 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.894935 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:09.895961 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.895938 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg"] Oct 31 03:56:09.905600 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:09.905573 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-86a834-d475474d9-vg2sg"] Oct 31 03:56:10.097759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:10.097566 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:10.249867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:10.249831 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:10.402046 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:10.401952 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:10.708740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:10.708701 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:11.017763 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:11.014568 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:11.579751 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:11.579714 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8353165-d9b4-445a-9a11-5fdf04307860" path="/var/lib/kubelet/pods/b8353165-d9b4-445a-9a11-5fdf04307860/volumes" Oct 31 03:56:11.611622 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:11.606526 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7"] Oct 31 03:56:11.611622 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:11.606799 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" podUID="253719d9-0fac-4d41-bfc3-d4d359ba04fe" containerName="maas-api" containerID="cri-o://46fbe8f11e4bf3fd2ead6aa0fbca1a5a6ae4a0e07d6ee6ea419473607dbf493e" gracePeriod=30 Oct 31 03:56:11.693756 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:11.693715 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:11.919641 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:11.919602 5490 generic.go:356] "Generic (PLEG): container finished" podID="253719d9-0fac-4d41-bfc3-d4d359ba04fe" containerID="46fbe8f11e4bf3fd2ead6aa0fbca1a5a6ae4a0e07d6ee6ea419473607dbf493e" exitCode=0 Oct 31 03:56:11.920092 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:11.920037 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" event={"ID":"253719d9-0fac-4d41-bfc3-d4d359ba04fe","Type":"ContainerDied","Data":"46fbe8f11e4bf3fd2ead6aa0fbca1a5a6ae4a0e07d6ee6ea419473607dbf493e"} Oct 31 03:56:11.951974 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:11.951661 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:12.036992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.036960 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" Oct 31 03:56:12.059317 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.059280 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/253719d9-0fac-4d41-bfc3-d4d359ba04fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/253719d9-0fac-4d41-bfc3-d4d359ba04fe-openshift-service-ca\") pod \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\" (UID: \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\") " Oct 31 03:56:12.059526 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.059382 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253719d9-0fac-4d41-bfc3-d4d359ba04fe-kube-api-access-x854h\" (UniqueName: \"kubernetes.io/projected/253719d9-0fac-4d41-bfc3-d4d359ba04fe-kube-api-access-x854h\") pod \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\" (UID: \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\") " Oct 31 03:56:12.059526 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.059512 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/253719d9-0fac-4d41-bfc3-d4d359ba04fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/253719d9-0fac-4d41-bfc3-d4d359ba04fe-metrics-certs\") pod \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\" (UID: \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\") " Oct 31 03:56:12.059647 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.059630 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/253719d9-0fac-4d41-bfc3-d4d359ba04fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/253719d9-0fac-4d41-bfc3-d4d359ba04fe-maas-api-tls\") pod \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\" (UID: \"253719d9-0fac-4d41-bfc3-d4d359ba04fe\") " Oct 31 03:56:12.059761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.059735 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/253719d9-0fac-4d41-bfc3-d4d359ba04fe-openshift-service-ca" pod "253719d9-0fac-4d41-bfc3-d4d359ba04fe" (UID: "253719d9-0fac-4d41-bfc3-d4d359ba04fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:12.060211 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.060186 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/253719d9-0fac-4d41-bfc3-d4d359ba04fe-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:12.063658 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.063629 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253719d9-0fac-4d41-bfc3-d4d359ba04fe-kube-api-access-x854h" pod "253719d9-0fac-4d41-bfc3-d4d359ba04fe" (UID: "253719d9-0fac-4d41-bfc3-d4d359ba04fe"). InnerVolumeSpecName "kube-api-access-x854h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:12.063822 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.063803 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/253719d9-0fac-4d41-bfc3-d4d359ba04fe-metrics-certs" pod "253719d9-0fac-4d41-bfc3-d4d359ba04fe" (UID: "253719d9-0fac-4d41-bfc3-d4d359ba04fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:12.064717 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.064689 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/253719d9-0fac-4d41-bfc3-d4d359ba04fe-maas-api-tls" pod "253719d9-0fac-4d41-bfc3-d4d359ba04fe" (UID: "253719d9-0fac-4d41-bfc3-d4d359ba04fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:12.092996 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.092954 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:12.161477 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.161432 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x854h\" (UniqueName: \"kubernetes.io/projected/253719d9-0fac-4d41-bfc3-d4d359ba04fe-kube-api-access-x854h\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:12.161477 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.161465 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/253719d9-0fac-4d41-bfc3-d4d359ba04fe-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:12.161477 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.161477 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/253719d9-0fac-4d41-bfc3-d4d359ba04fe-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:12.238555 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.238457 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:12.609522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.609437 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:12.928951 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.928920 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" Oct 31 03:56:12.929134 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.928954 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7" event={"ID":"253719d9-0fac-4d41-bfc3-d4d359ba04fe","Type":"ContainerDied","Data":"249edabfff48031be2a97babfb8fc95280527b8bfa22cd703eb79f1188bcbb19"} Oct 31 03:56:12.929134 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.929023 5490 scope.go:122] "RemoveContainer" containerID="46fbe8f11e4bf3fd2ead6aa0fbca1a5a6ae4a0e07d6ee6ea419473607dbf493e" Oct 31 03:56:12.956056 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.956024 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7"] Oct 31 03:56:12.960320 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:12.960297 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ce9b89-5745885799-w28x7"] Oct 31 03:56:13.001621 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:13.001572 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:13.189615 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:13.189525 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:13.479870 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:13.479785 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:13.559356 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:13.559318 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996"] Oct 31 03:56:13.559936 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:13.559902 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" podUID="1ce35023-5f08-4356-8e09-4328952c6040" containerName="istio-proxy" containerID="cri-o://de079737f212ac8275e31adf6ab22910b8efefe3c4d6bd57862bdc43663ad8b4" gracePeriod=30 Oct 31 03:56:13.578135 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:13.578109 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="253719d9-0fac-4d41-bfc3-d4d359ba04fe" path="/var/lib/kubelet/pods/253719d9-0fac-4d41-bfc3-d4d359ba04fe/volumes" Oct 31 03:56:13.631989 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:13.631949 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:14.016381 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:14.016339 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:14.199739 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:14.199704 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:15.030163 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:15.030081 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:15.819552 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:15.819084 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:16.173217 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:16.173174 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:16.528574 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:16.528487 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:16.684974 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:16.684915 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:16.899730 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:16.899630 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:17.162594 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:17.162557 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:56:17.310875 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:17.310820 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:17.621201 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:17.621114 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:18.081834 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:18.081586 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:18.332609 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:18.332517 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf"] Oct 31 03:56:18.333045 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:18.332808 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" podUID="01e9fcdd-b435-47dc-a064-02ff1d074156" containerName="istio-proxy" containerID="cri-o://634cf9cedaca6b5617d83f48568ecf61f6d274080ab6f25114cbe918b4fb6ca8" gracePeriod=30 Oct 31 03:56:18.338215 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:18.338189 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45"] Oct 31 03:56:18.338440 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:18.338416 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" podUID="baea2599-5723-4c5e-b063-e960ec09e121" containerName="istio-proxy" containerID="cri-o://23ad0c13c66bb7005458fde49c3ab6eb81c1bd5c866c0a93f6e33b4dd94cadda" gracePeriod=30 Oct 31 03:56:18.351064 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:18.351029 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:18.509728 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:18.509316 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:18.662265 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:18.662226 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:18.870389 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:18.870312 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:56:18.992267 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:18.992228 5490 generic.go:356] "Generic (PLEG): container finished" podID="1ce35023-5f08-4356-8e09-4328952c6040" containerID="de079737f212ac8275e31adf6ab22910b8efefe3c4d6bd57862bdc43663ad8b4" exitCode=0 Oct 31 03:56:18.992464 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:18.992325 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" Oct 31 03:56:18.992464 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:18.992326 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" event={"ID":"1ce35023-5f08-4356-8e09-4328952c6040","Type":"ContainerDied","Data":"de079737f212ac8275e31adf6ab22910b8efefe3c4d6bd57862bdc43663ad8b4"} Oct 31 03:56:18.992464 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:18.992395 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996" event={"ID":"1ce35023-5f08-4356-8e09-4328952c6040","Type":"ContainerDied","Data":"198ccfc0959cf08c3a48f73203ee530c01bfe908ad3369b0efe34dbdb0b3b210"} Oct 31 03:56:18.992464 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:18.992424 5490 scope.go:122] "RemoveContainer" containerID="de079737f212ac8275e31adf6ab22910b8efefe3c4d6bd57862bdc43663ad8b4" Oct 31 03:56:18.995958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:18.995937 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj"] Oct 31 03:56:19.001241 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.001205 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ce35023-5f08-4356-8e09-4328952c6040-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce35023-5f08-4356-8e09-4328952c6040-istio-podinfo\") pod \"1ce35023-5f08-4356-8e09-4328952c6040\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " Oct 31 03:56:19.001403 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.001262 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-istio-data\") pod \"1ce35023-5f08-4356-8e09-4328952c6040\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " Oct 31 03:56:19.001403 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.001289 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-credential-socket\") pod \"1ce35023-5f08-4356-8e09-4328952c6040\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " Oct 31 03:56:19.001526 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.001434 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce35023-5f08-4356-8e09-4328952c6040-istio-token\" (UniqueName: \"kubernetes.io/projected/1ce35023-5f08-4356-8e09-4328952c6040-istio-token\") pod \"1ce35023-5f08-4356-8e09-4328952c6040\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " Oct 31 03:56:19.001526 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.001470 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-istio-envoy\") pod \"1ce35023-5f08-4356-8e09-4328952c6040\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " Oct 31 03:56:19.001621 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.001528 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-workload-certs\") pod \"1ce35023-5f08-4356-8e09-4328952c6040\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " Oct 31 03:56:19.001621 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.001590 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce35023-5f08-4356-8e09-4328952c6040-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce35023-5f08-4356-8e09-4328952c6040-istiod-ca-cert\") pod \"1ce35023-5f08-4356-8e09-4328952c6040\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " Oct 31 03:56:19.001834 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.001665 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-workload-socket\") pod \"1ce35023-5f08-4356-8e09-4328952c6040\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " Oct 31 03:56:19.003889 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.003864 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce35023-5f08-4356-8e09-4328952c6040-kube-api-access-h652q\" (UniqueName: \"kubernetes.io/projected/1ce35023-5f08-4356-8e09-4328952c6040-kube-api-access-h652q\") pod \"1ce35023-5f08-4356-8e09-4328952c6040\" (UID: \"1ce35023-5f08-4356-8e09-4328952c6040\") " Oct 31 03:56:19.004088 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.001859 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-credential-socket" pod "1ce35023-5f08-4356-8e09-4328952c6040" (UID: "1ce35023-5f08-4356-8e09-4328952c6040"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:19.004168 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.002127 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-istio-data" pod "1ce35023-5f08-4356-8e09-4328952c6040" (UID: "1ce35023-5f08-4356-8e09-4328952c6040"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:19.004168 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.002159 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce35023-5f08-4356-8e09-4328952c6040-istiod-ca-cert" pod "1ce35023-5f08-4356-8e09-4328952c6040" (UID: "1ce35023-5f08-4356-8e09-4328952c6040"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:19.004168 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.002327 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-workload-socket" pod "1ce35023-5f08-4356-8e09-4328952c6040" (UID: "1ce35023-5f08-4356-8e09-4328952c6040"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:19.004168 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.002322 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-workload-certs" pod "1ce35023-5f08-4356-8e09-4328952c6040" (UID: "1ce35023-5f08-4356-8e09-4328952c6040"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:19.004987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.004667 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:19.005091 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.005004 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:19.005091 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.005030 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:19.005091 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.005044 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce35023-5f08-4356-8e09-4328952c6040-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:19.005091 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.005059 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:19.005091 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.004901 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-istio-envoy" pod "1ce35023-5f08-4356-8e09-4328952c6040" (UID: "1ce35023-5f08-4356-8e09-4328952c6040"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:19.005091 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.004950 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce35023-5f08-4356-8e09-4328952c6040-istio-token" pod "1ce35023-5f08-4356-8e09-4328952c6040" (UID: "1ce35023-5f08-4356-8e09-4328952c6040"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:19.005643 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.005621 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ce35023-5f08-4356-8e09-4328952c6040-istio-podinfo" pod "1ce35023-5f08-4356-8e09-4328952c6040" (UID: "1ce35023-5f08-4356-8e09-4328952c6040"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:19.006861 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.006839 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6"] Oct 31 03:56:19.008506 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.008413 5490 scope.go:122] "RemoveContainer" containerID="07728346be99c829fa088b863aa13c68d084522ef6b407b64fc2ccdf4ad2211e" Oct 31 03:56:19.019794 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.019765 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce35023-5f08-4356-8e09-4328952c6040-kube-api-access-h652q" pod "1ce35023-5f08-4356-8e09-4328952c6040" (UID: "1ce35023-5f08-4356-8e09-4328952c6040"). InnerVolumeSpecName "kube-api-access-h652q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:19.048409 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.048376 5490 scope.go:122] "RemoveContainer" containerID="de079737f212ac8275e31adf6ab22910b8efefe3c4d6bd57862bdc43663ad8b4" Oct 31 03:56:19.048938 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:19.048914 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de079737f212ac8275e31adf6ab22910b8efefe3c4d6bd57862bdc43663ad8b4\": container with ID starting with de079737f212ac8275e31adf6ab22910b8efefe3c4d6bd57862bdc43663ad8b4 not found: ID does not exist" containerID="de079737f212ac8275e31adf6ab22910b8efefe3c4d6bd57862bdc43663ad8b4" Oct 31 03:56:19.049038 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.048945 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de079737f212ac8275e31adf6ab22910b8efefe3c4d6bd57862bdc43663ad8b4"} err="failed to get container status \"de079737f212ac8275e31adf6ab22910b8efefe3c4d6bd57862bdc43663ad8b4\": rpc error: code = NotFound desc = could not find container \"de079737f212ac8275e31adf6ab22910b8efefe3c4d6bd57862bdc43663ad8b4\": container with ID starting with de079737f212ac8275e31adf6ab22910b8efefe3c4d6bd57862bdc43663ad8b4 not found: ID does not exist" Oct 31 03:56:19.049038 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.048977 5490 scope.go:122] "RemoveContainer" containerID="07728346be99c829fa088b863aa13c68d084522ef6b407b64fc2ccdf4ad2211e" Oct 31 03:56:19.049274 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:19.049260 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07728346be99c829fa088b863aa13c68d084522ef6b407b64fc2ccdf4ad2211e\": container with ID starting with 07728346be99c829fa088b863aa13c68d084522ef6b407b64fc2ccdf4ad2211e not found: ID does not exist" containerID="07728346be99c829fa088b863aa13c68d084522ef6b407b64fc2ccdf4ad2211e" Oct 31 03:56:19.049312 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.049279 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07728346be99c829fa088b863aa13c68d084522ef6b407b64fc2ccdf4ad2211e"} err="failed to get container status \"07728346be99c829fa088b863aa13c68d084522ef6b407b64fc2ccdf4ad2211e\": rpc error: code = NotFound desc = could not find container \"07728346be99c829fa088b863aa13c68d084522ef6b407b64fc2ccdf4ad2211e\": container with ID starting with 07728346be99c829fa088b863aa13c68d084522ef6b407b64fc2ccdf4ad2211e not found: ID does not exist" Oct 31 03:56:19.059275 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.059237 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:19.106167 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.106127 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce35023-5f08-4356-8e09-4328952c6040-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:19.106167 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.106161 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce35023-5f08-4356-8e09-4328952c6040-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:19.106167 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.106173 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h652q\" (UniqueName: \"kubernetes.io/projected/1ce35023-5f08-4356-8e09-4328952c6040-kube-api-access-h652q\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:19.106421 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.106184 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce35023-5f08-4356-8e09-4328952c6040-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:19.332869 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.332833 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996"] Oct 31 03:56:19.340595 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.340552 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-p5996"] Oct 31 03:56:19.358451 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.358410 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:19.407808 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.407760 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5"] Oct 31 03:56:19.408425 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.408387 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" podUID="dacc99d6-e9da-46e0-841d-14c5f54dd8fa" containerName="istio-proxy" containerID="cri-o://0998ecfcad5304891c2a3145d4d7a7e940ec5d1406fc4f5311365b9172a45253" gracePeriod=30 Oct 31 03:56:19.410943 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.410915 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg"] Oct 31 03:56:19.411571 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.411520 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" podUID="209cb838-5556-47a9-96ae-e2378ad8c8b5" containerName="istio-proxy" containerID="cri-o://695ae0d222a4ce1f614904f66c785fb60a236e16e77f07fb2a54bde17bcc8031" gracePeriod=30 Oct 31 03:56:19.509974 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.509810 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:19.586310 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.586217 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce35023-5f08-4356-8e09-4328952c6040" path="/var/lib/kubelet/pods/1ce35023-5f08-4356-8e09-4328952c6040/volumes" Oct 31 03:56:19.805636 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:19.805600 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:20.004560 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:20.004522 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:20.920230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:20.920182 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:21.349784 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:21.349635 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:56:21.349976 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:21.349770 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" podUID="01e9fcdd-b435-47dc-a064-02ff1d074156" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:56:21.364666 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:21.364631 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:21.536895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:21.527093 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:21.536895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:21.528893 5490 patch_prober.go:29] interesting pod/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:56:21.536895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:21.528958 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" podUID="209cb838-5556-47a9-96ae-e2378ad8c8b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:56:21.536895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:21.529320 5490 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:56:21.536895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:21.529366 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" podUID="baea2599-5723-4c5e-b063-e960ec09e121" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:56:21.680100 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:21.680056 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:21.854810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:21.854773 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:22.171813 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:22.170085 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd"] Oct 31 03:56:22.171813 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:22.170383 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" podUID="05bbcb81-55ce-4387-89c3-f6350fe8636f" containerName="istio-proxy" containerID="cri-o://ce4f1160f273d64b012eb62dba9a79a15defc4613a3b6e8efbfe985e6ec2d6d0" gracePeriod=30 Oct 31 03:56:22.224732 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:22.224644 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:22.229132 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:22.228801 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl"] Oct 31 03:56:22.229410 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:22.229377 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" podUID="824618ba-ce21-4eeb-bf27-f1a861327863" containerName="istio-proxy" containerID="cri-o://41f67ea9dd6fb51c2bb5ab18a755f9a86020168a384eca37e881b3afe0bceb47" gracePeriod=30 Oct 31 03:56:22.296251 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:22.296205 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h"] Oct 31 03:56:22.297243 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:22.297203 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" podUID="8317b14b-20f5-4404-8f4c-bf315ec53736" containerName="istio-proxy" containerID="cri-o://75d33ed8265ccc3c5cd1979a992cbb481d894e7d6715fc035d67fb91b327dccf" gracePeriod=30 Oct 31 03:56:22.313828 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:22.313792 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh"] Oct 31 03:56:22.314774 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:22.314742 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" podUID="f57cc204-16b4-4735-82c1-7cee96c1704d" containerName="istio-proxy" containerID="cri-o://977556c538ae57240b5b254d28dbffae276a3566ad2bf1217ca04aa05d4bf3b0" gracePeriod=30 Oct 31 03:56:22.325857 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:22.325480 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:56:22.325857 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:22.325534 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" podUID="824618ba-ce21-4eeb-bf27-f1a861327863" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:56:22.371552 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:22.370056 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:22.433379 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:22.433285 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:56:22.433379 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:22.433340 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" podUID="05bbcb81-55ce-4387-89c3-f6350fe8636f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:56:22.502610 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:22.502570 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5"] Oct 31 03:56:22.520670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:22.520632 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll"] Oct 31 03:56:22.711258 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:22.711179 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:23.684763 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.684731 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:56:23.753080 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.753051 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:56:23.781644 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.781609 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-credential-socket\") pod \"baea2599-5723-4c5e-b063-e960ec09e121\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " Oct 31 03:56:23.781853 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.781751 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baea2599-5723-4c5e-b063-e960ec09e121-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baea2599-5723-4c5e-b063-e960ec09e121-istiod-ca-cert\") pod \"baea2599-5723-4c5e-b063-e960ec09e121\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " Oct 31 03:56:23.781853 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.781795 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-istio-data\") pod \"baea2599-5723-4c5e-b063-e960ec09e121\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " Oct 31 03:56:23.781979 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.781864 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baea2599-5723-4c5e-b063-e960ec09e121-istio-token\" (UniqueName: \"kubernetes.io/projected/baea2599-5723-4c5e-b063-e960ec09e121-istio-token\") pod \"baea2599-5723-4c5e-b063-e960ec09e121\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " Oct 31 03:56:23.781979 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.781897 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/baea2599-5723-4c5e-b063-e960ec09e121-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baea2599-5723-4c5e-b063-e960ec09e121-istio-podinfo\") pod \"baea2599-5723-4c5e-b063-e960ec09e121\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " Oct 31 03:56:23.781979 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.781961 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-workload-certs\") pod \"baea2599-5723-4c5e-b063-e960ec09e121\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " Oct 31 03:56:23.782165 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.781992 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-istio-envoy\") pod \"baea2599-5723-4c5e-b063-e960ec09e121\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " Oct 31 03:56:23.782165 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.781986 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-credential-socket" pod "baea2599-5723-4c5e-b063-e960ec09e121" (UID: "baea2599-5723-4c5e-b063-e960ec09e121"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:23.782165 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.782126 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baea2599-5723-4c5e-b063-e960ec09e121-kube-api-access-smhgj\" (UniqueName: \"kubernetes.io/projected/baea2599-5723-4c5e-b063-e960ec09e121-kube-api-access-smhgj\") pod \"baea2599-5723-4c5e-b063-e960ec09e121\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " Oct 31 03:56:23.782401 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.782193 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-istio-data" pod "baea2599-5723-4c5e-b063-e960ec09e121" (UID: "baea2599-5723-4c5e-b063-e960ec09e121"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:23.782401 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.782201 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-workload-socket\") pod \"baea2599-5723-4c5e-b063-e960ec09e121\" (UID: \"baea2599-5723-4c5e-b063-e960ec09e121\") " Oct 31 03:56:23.782401 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.782229 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baea2599-5723-4c5e-b063-e960ec09e121-istiod-ca-cert" pod "baea2599-5723-4c5e-b063-e960ec09e121" (UID: "baea2599-5723-4c5e-b063-e960ec09e121"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:23.782638 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.782459 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-workload-socket" pod "baea2599-5723-4c5e-b063-e960ec09e121" (UID: "baea2599-5723-4c5e-b063-e960ec09e121"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:23.783086 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.783059 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baea2599-5723-4c5e-b063-e960ec09e121-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.783200 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.783093 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.783200 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.783111 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.783200 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.783128 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.784178 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.784154 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-workload-certs" pod "baea2599-5723-4c5e-b063-e960ec09e121" (UID: "baea2599-5723-4c5e-b063-e960ec09e121"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:23.784336 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.784317 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baea2599-5723-4c5e-b063-e960ec09e121-istio-token" pod "baea2599-5723-4c5e-b063-e960ec09e121" (UID: "baea2599-5723-4c5e-b063-e960ec09e121"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:23.784408 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.784319 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/baea2599-5723-4c5e-b063-e960ec09e121-istio-podinfo" pod "baea2599-5723-4c5e-b063-e960ec09e121" (UID: "baea2599-5723-4c5e-b063-e960ec09e121"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:23.784598 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.784585 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-istio-envoy" pod "baea2599-5723-4c5e-b063-e960ec09e121" (UID: "baea2599-5723-4c5e-b063-e960ec09e121"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:23.784643 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.784607 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baea2599-5723-4c5e-b063-e960ec09e121-kube-api-access-smhgj" pod "baea2599-5723-4c5e-b063-e960ec09e121" (UID: "baea2599-5723-4c5e-b063-e960ec09e121"). InnerVolumeSpecName "kube-api-access-smhgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:23.886246 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.886151 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-workload-certs\") pod \"01e9fcdd-b435-47dc-a064-02ff1d074156\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " Oct 31 03:56:23.886246 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.886230 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-envoy\") pod \"01e9fcdd-b435-47dc-a064-02ff1d074156\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " Oct 31 03:56:23.886504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.886263 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-workload-socket\") pod \"01e9fcdd-b435-47dc-a064-02ff1d074156\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " Oct 31 03:56:23.886504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.886327 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-podinfo\") pod \"01e9fcdd-b435-47dc-a064-02ff1d074156\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " Oct 31 03:56:23.886504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.886424 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-data\") pod \"01e9fcdd-b435-47dc-a064-02ff1d074156\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " Oct 31 03:56:23.886504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.886461 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01e9fcdd-b435-47dc-a064-02ff1d074156-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01e9fcdd-b435-47dc-a064-02ff1d074156-istiod-ca-cert\") pod \"01e9fcdd-b435-47dc-a064-02ff1d074156\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " Oct 31 03:56:23.886504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.886490 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-token\" (UniqueName: \"kubernetes.io/projected/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-token\") pod \"01e9fcdd-b435-47dc-a064-02ff1d074156\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " Oct 31 03:56:23.886856 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.886582 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-credential-socket\") pod \"01e9fcdd-b435-47dc-a064-02ff1d074156\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " Oct 31 03:56:23.886856 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.886696 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01e9fcdd-b435-47dc-a064-02ff1d074156-kube-api-access-7snbb\" (UniqueName: \"kubernetes.io/projected/01e9fcdd-b435-47dc-a064-02ff1d074156-kube-api-access-7snbb\") pod \"01e9fcdd-b435-47dc-a064-02ff1d074156\" (UID: \"01e9fcdd-b435-47dc-a064-02ff1d074156\") " Oct 31 03:56:23.887457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.887428 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baea2599-5723-4c5e-b063-e960ec09e121-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.887457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.887456 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baea2599-5723-4c5e-b063-e960ec09e121-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.887660 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.887472 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.887660 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.887488 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baea2599-5723-4c5e-b063-e960ec09e121-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.887660 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.887506 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smhgj\" (UniqueName: \"kubernetes.io/projected/baea2599-5723-4c5e-b063-e960ec09e121-kube-api-access-smhgj\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.890519 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.890492 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-podinfo" pod "01e9fcdd-b435-47dc-a064-02ff1d074156" (UID: "01e9fcdd-b435-47dc-a064-02ff1d074156"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:23.890803 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.890769 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-workload-certs" pod "01e9fcdd-b435-47dc-a064-02ff1d074156" (UID: "01e9fcdd-b435-47dc-a064-02ff1d074156"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:23.892741 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.892659 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-envoy" pod "01e9fcdd-b435-47dc-a064-02ff1d074156" (UID: "01e9fcdd-b435-47dc-a064-02ff1d074156"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:23.893015 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.892997 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-workload-socket" pod "01e9fcdd-b435-47dc-a064-02ff1d074156" (UID: "01e9fcdd-b435-47dc-a064-02ff1d074156"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:23.894114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.894069 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-data" pod "01e9fcdd-b435-47dc-a064-02ff1d074156" (UID: "01e9fcdd-b435-47dc-a064-02ff1d074156"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:23.894300 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.894276 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-credential-socket" pod "01e9fcdd-b435-47dc-a064-02ff1d074156" (UID: "01e9fcdd-b435-47dc-a064-02ff1d074156"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:23.894474 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.894456 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e9fcdd-b435-47dc-a064-02ff1d074156-kube-api-access-7snbb" pod "01e9fcdd-b435-47dc-a064-02ff1d074156" (UID: "01e9fcdd-b435-47dc-a064-02ff1d074156"). InnerVolumeSpecName "kube-api-access-7snbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:23.894814 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.894794 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e9fcdd-b435-47dc-a064-02ff1d074156-istiod-ca-cert" pod "01e9fcdd-b435-47dc-a064-02ff1d074156" (UID: "01e9fcdd-b435-47dc-a064-02ff1d074156"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:23.895194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.895174 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-token" pod "01e9fcdd-b435-47dc-a064-02ff1d074156" (UID: "01e9fcdd-b435-47dc-a064-02ff1d074156"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:23.958184 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.958141 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:23.989016 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.988976 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.989016 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.989014 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7snbb\" (UniqueName: \"kubernetes.io/projected/01e9fcdd-b435-47dc-a064-02ff1d074156-kube-api-access-7snbb\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.989016 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.989026 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.989299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.989035 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.989299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.989044 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.989299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.989053 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.989299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.989061 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.989299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.989069 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01e9fcdd-b435-47dc-a064-02ff1d074156-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:23.989299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:23.989078 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01e9fcdd-b435-47dc-a064-02ff1d074156-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:24.038190 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.038152 5490 generic.go:356] "Generic (PLEG): container finished" podID="baea2599-5723-4c5e-b063-e960ec09e121" containerID="23ad0c13c66bb7005458fde49c3ab6eb81c1bd5c866c0a93f6e33b4dd94cadda" exitCode=0 Oct 31 03:56:24.038364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.038202 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" Oct 31 03:56:24.038364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.038237 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" event={"ID":"baea2599-5723-4c5e-b063-e960ec09e121","Type":"ContainerDied","Data":"23ad0c13c66bb7005458fde49c3ab6eb81c1bd5c866c0a93f6e33b4dd94cadda"} Oct 31 03:56:24.038364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.038296 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45" event={"ID":"baea2599-5723-4c5e-b063-e960ec09e121","Type":"ContainerDied","Data":"fda7b4800aa5ce3a3dec477b2dfad9b1dff5c5df99ed920772d3f63ec335cada"} Oct 31 03:56:24.038364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.038331 5490 scope.go:122] "RemoveContainer" containerID="23ad0c13c66bb7005458fde49c3ab6eb81c1bd5c866c0a93f6e33b4dd94cadda" Oct 31 03:56:24.040043 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.040015 5490 generic.go:356] "Generic (PLEG): container finished" podID="01e9fcdd-b435-47dc-a064-02ff1d074156" containerID="634cf9cedaca6b5617d83f48568ecf61f6d274080ab6f25114cbe918b4fb6ca8" exitCode=0 Oct 31 03:56:24.040189 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.040062 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" event={"ID":"01e9fcdd-b435-47dc-a064-02ff1d074156","Type":"ContainerDied","Data":"634cf9cedaca6b5617d83f48568ecf61f6d274080ab6f25114cbe918b4fb6ca8"} Oct 31 03:56:24.040189 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.040072 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" Oct 31 03:56:24.040189 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.040102 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf" event={"ID":"01e9fcdd-b435-47dc-a064-02ff1d074156","Type":"ContainerDied","Data":"e1488f3cfaaa14b6baec0bd90795cdcfa306573e7cc97c9105e31e58162cb94e"} Oct 31 03:56:24.056565 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.055454 5490 scope.go:122] "RemoveContainer" containerID="23ad0c13c66bb7005458fde49c3ab6eb81c1bd5c866c0a93f6e33b4dd94cadda" Oct 31 03:56:24.056565 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:24.056496 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23ad0c13c66bb7005458fde49c3ab6eb81c1bd5c866c0a93f6e33b4dd94cadda\": container with ID starting with 23ad0c13c66bb7005458fde49c3ab6eb81c1bd5c866c0a93f6e33b4dd94cadda not found: ID does not exist" containerID="23ad0c13c66bb7005458fde49c3ab6eb81c1bd5c866c0a93f6e33b4dd94cadda" Oct 31 03:56:24.056565 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.056529 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ad0c13c66bb7005458fde49c3ab6eb81c1bd5c866c0a93f6e33b4dd94cadda"} err="failed to get container status \"23ad0c13c66bb7005458fde49c3ab6eb81c1bd5c866c0a93f6e33b4dd94cadda\": rpc error: code = NotFound desc = could not find container \"23ad0c13c66bb7005458fde49c3ab6eb81c1bd5c866c0a93f6e33b4dd94cadda\": container with ID starting with 23ad0c13c66bb7005458fde49c3ab6eb81c1bd5c866c0a93f6e33b4dd94cadda not found: ID does not exist" Oct 31 03:56:24.056565 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.056547 5490 scope.go:122] "RemoveContainer" containerID="634cf9cedaca6b5617d83f48568ecf61f6d274080ab6f25114cbe918b4fb6ca8" Oct 31 03:56:24.068256 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.068227 5490 scope.go:122] "RemoveContainer" containerID="634cf9cedaca6b5617d83f48568ecf61f6d274080ab6f25114cbe918b4fb6ca8" Oct 31 03:56:24.068585 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:24.068561 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"634cf9cedaca6b5617d83f48568ecf61f6d274080ab6f25114cbe918b4fb6ca8\": container with ID starting with 634cf9cedaca6b5617d83f48568ecf61f6d274080ab6f25114cbe918b4fb6ca8 not found: ID does not exist" containerID="634cf9cedaca6b5617d83f48568ecf61f6d274080ab6f25114cbe918b4fb6ca8" Oct 31 03:56:24.068655 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.068595 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"634cf9cedaca6b5617d83f48568ecf61f6d274080ab6f25114cbe918b4fb6ca8"} err="failed to get container status \"634cf9cedaca6b5617d83f48568ecf61f6d274080ab6f25114cbe918b4fb6ca8\": rpc error: code = NotFound desc = could not find container \"634cf9cedaca6b5617d83f48568ecf61f6d274080ab6f25114cbe918b4fb6ca8\": container with ID starting with 634cf9cedaca6b5617d83f48568ecf61f6d274080ab6f25114cbe918b4fb6ca8 not found: ID does not exist" Oct 31 03:56:24.078780 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.078749 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45"] Oct 31 03:56:24.083249 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.083215 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-wxx45"] Oct 31 03:56:24.090709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.090664 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf"] Oct 31 03:56:24.097957 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.097924 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-q8xrf"] Oct 31 03:56:24.421326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.421286 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2"] Oct 31 03:56:24.449326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.449288 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv"] Oct 31 03:56:24.548504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.548472 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:24.689985 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.689099 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:24.806717 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.802078 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:56:24.861441 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.861408 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:56:24.924158 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.924104 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istiod-ca-cert\") pod \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " Oct 31 03:56:24.924351 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.924197 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-workload-socket\") pod \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " Oct 31 03:56:24.924351 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.924300 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-kube-api-access-ljn8x\" (UniqueName: \"kubernetes.io/projected/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-kube-api-access-ljn8x\") pod \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " Oct 31 03:56:24.924351 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.924339 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-podinfo\") pod \"209cb838-5556-47a9-96ae-e2378ad8c8b5\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " Oct 31 03:56:24.924564 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.924398 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-envoy\") pod \"209cb838-5556-47a9-96ae-e2378ad8c8b5\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " Oct 31 03:56:24.924564 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.924537 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-data\") pod \"209cb838-5556-47a9-96ae-e2378ad8c8b5\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " Oct 31 03:56:24.924709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.924605 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-workload-certs\") pod \"209cb838-5556-47a9-96ae-e2378ad8c8b5\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " Oct 31 03:56:24.924709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.924619 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istiod-ca-cert" pod "dacc99d6-e9da-46e0-841d-14c5f54dd8fa" (UID: "dacc99d6-e9da-46e0-841d-14c5f54dd8fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:24.924709 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.924701 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/209cb838-5556-47a9-96ae-e2378ad8c8b5-kube-api-access-7tr8z\" (UniqueName: \"kubernetes.io/projected/209cb838-5556-47a9-96ae-e2378ad8c8b5-kube-api-access-7tr8z\") pod \"209cb838-5556-47a9-96ae-e2378ad8c8b5\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " Oct 31 03:56:24.925475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.924750 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-workload-socket" pod "dacc99d6-e9da-46e0-841d-14c5f54dd8fa" (UID: "dacc99d6-e9da-46e0-841d-14c5f54dd8fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:24.925475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.924763 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-podinfo\") pod \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " Oct 31 03:56:24.925475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.924838 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-credential-socket\") pod \"209cb838-5556-47a9-96ae-e2378ad8c8b5\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " Oct 31 03:56:24.925475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.924890 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-data\") pod \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " Oct 31 03:56:24.925475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.924962 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-credential-socket\") pod \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " Oct 31 03:56:24.925475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.925007 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-workload-certs" pod "209cb838-5556-47a9-96ae-e2378ad8c8b5" (UID: "209cb838-5556-47a9-96ae-e2378ad8c8b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:24.925475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.925039 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-envoy\") pod \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " Oct 31 03:56:24.925475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.925109 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-workload-socket\") pod \"209cb838-5556-47a9-96ae-e2378ad8c8b5\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " Oct 31 03:56:24.925475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.925172 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-token\" (UniqueName: \"kubernetes.io/projected/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-token\") pod \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " Oct 31 03:56:24.925475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.925207 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-workload-certs\") pod \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\" (UID: \"dacc99d6-e9da-46e0-841d-14c5f54dd8fa\") " Oct 31 03:56:24.925475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.925245 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/209cb838-5556-47a9-96ae-e2378ad8c8b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209cb838-5556-47a9-96ae-e2378ad8c8b5-istiod-ca-cert\") pod \"209cb838-5556-47a9-96ae-e2378ad8c8b5\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " Oct 31 03:56:24.925475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.925284 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-token\" (UniqueName: \"kubernetes.io/projected/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-token\") pod \"209cb838-5556-47a9-96ae-e2378ad8c8b5\" (UID: \"209cb838-5556-47a9-96ae-e2378ad8c8b5\") " Oct 31 03:56:24.926319 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.925803 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-data" pod "dacc99d6-e9da-46e0-841d-14c5f54dd8fa" (UID: "dacc99d6-e9da-46e0-841d-14c5f54dd8fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:24.926319 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.925904 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-credential-socket" pod "dacc99d6-e9da-46e0-841d-14c5f54dd8fa" (UID: "dacc99d6-e9da-46e0-841d-14c5f54dd8fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:24.926319 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.926116 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:24.926319 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.926143 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:24.926319 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.926158 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:24.926319 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.926172 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:24.926319 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.926188 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:24.932234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.929702 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-token" pod "dacc99d6-e9da-46e0-841d-14c5f54dd8fa" (UID: "dacc99d6-e9da-46e0-841d-14c5f54dd8fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:24.932234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.930429 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-envoy" pod "dacc99d6-e9da-46e0-841d-14c5f54dd8fa" (UID: "dacc99d6-e9da-46e0-841d-14c5f54dd8fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:24.932234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.931756 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-data" pod "209cb838-5556-47a9-96ae-e2378ad8c8b5" (UID: "209cb838-5556-47a9-96ae-e2378ad8c8b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:24.932234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.931792 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-envoy" pod "209cb838-5556-47a9-96ae-e2378ad8c8b5" (UID: "209cb838-5556-47a9-96ae-e2378ad8c8b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:24.932234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.931906 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/209cb838-5556-47a9-96ae-e2378ad8c8b5-istiod-ca-cert" pod "209cb838-5556-47a9-96ae-e2378ad8c8b5" (UID: "209cb838-5556-47a9-96ae-e2378ad8c8b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:24.932815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.932429 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-workload-certs" pod "dacc99d6-e9da-46e0-841d-14c5f54dd8fa" (UID: "dacc99d6-e9da-46e0-841d-14c5f54dd8fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:24.933255 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.933200 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-workload-socket" pod "209cb838-5556-47a9-96ae-e2378ad8c8b5" (UID: "209cb838-5556-47a9-96ae-e2378ad8c8b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:24.933496 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.933476 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-credential-socket" pod "209cb838-5556-47a9-96ae-e2378ad8c8b5" (UID: "209cb838-5556-47a9-96ae-e2378ad8c8b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:24.936299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.936276 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-podinfo" pod "dacc99d6-e9da-46e0-841d-14c5f54dd8fa" (UID: "dacc99d6-e9da-46e0-841d-14c5f54dd8fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:24.936299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.936288 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-podinfo" pod "209cb838-5556-47a9-96ae-e2378ad8c8b5" (UID: "209cb838-5556-47a9-96ae-e2378ad8c8b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:24.936428 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.936288 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-kube-api-access-ljn8x" pod "dacc99d6-e9da-46e0-841d-14c5f54dd8fa" (UID: "dacc99d6-e9da-46e0-841d-14c5f54dd8fa"). InnerVolumeSpecName "kube-api-access-ljn8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:24.936428 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.936309 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-token" pod "209cb838-5556-47a9-96ae-e2378ad8c8b5" (UID: "209cb838-5556-47a9-96ae-e2378ad8c8b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:24.937493 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:24.937469 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209cb838-5556-47a9-96ae-e2378ad8c8b5-kube-api-access-7tr8z" pod "209cb838-5556-47a9-96ae-e2378ad8c8b5" (UID: "209cb838-5556-47a9-96ae-e2378ad8c8b5"). InnerVolumeSpecName "kube-api-access-7tr8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:25.027803 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.027763 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.027803 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.027800 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.028028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.027813 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tr8z\" (UniqueName: \"kubernetes.io/projected/209cb838-5556-47a9-96ae-e2378ad8c8b5-kube-api-access-7tr8z\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.028028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.027825 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.028028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.027834 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.028028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.027844 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.028028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.027853 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/209cb838-5556-47a9-96ae-e2378ad8c8b5-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.028028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.027861 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.028028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.027871 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.028028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.027882 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/209cb838-5556-47a9-96ae-e2378ad8c8b5-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.028028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.027890 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.028028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.027899 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljn8x\" (UniqueName: \"kubernetes.io/projected/dacc99d6-e9da-46e0-841d-14c5f54dd8fa-kube-api-access-ljn8x\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.028028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.027908 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/209cb838-5556-47a9-96ae-e2378ad8c8b5-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:25.051140 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.051106 5490 generic.go:356] "Generic (PLEG): container finished" podID="209cb838-5556-47a9-96ae-e2378ad8c8b5" containerID="695ae0d222a4ce1f614904f66c785fb60a236e16e77f07fb2a54bde17bcc8031" exitCode=0 Oct 31 03:56:25.051359 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.051158 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" Oct 31 03:56:25.051359 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.051190 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" event={"ID":"209cb838-5556-47a9-96ae-e2378ad8c8b5","Type":"ContainerDied","Data":"695ae0d222a4ce1f614904f66c785fb60a236e16e77f07fb2a54bde17bcc8031"} Oct 31 03:56:25.051359 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.051256 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg" event={"ID":"209cb838-5556-47a9-96ae-e2378ad8c8b5","Type":"ContainerDied","Data":"9045f4a5ec70a0b9774d36565b6e367240631f7891f76d078371fe92710b168b"} Oct 31 03:56:25.051359 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.051291 5490 scope.go:122] "RemoveContainer" containerID="695ae0d222a4ce1f614904f66c785fb60a236e16e77f07fb2a54bde17bcc8031" Oct 31 03:56:25.052918 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.052892 5490 generic.go:356] "Generic (PLEG): container finished" podID="dacc99d6-e9da-46e0-841d-14c5f54dd8fa" containerID="0998ecfcad5304891c2a3145d4d7a7e940ec5d1406fc4f5311365b9172a45253" exitCode=0 Oct 31 03:56:25.053051 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.052987 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" event={"ID":"dacc99d6-e9da-46e0-841d-14c5f54dd8fa","Type":"ContainerDied","Data":"0998ecfcad5304891c2a3145d4d7a7e940ec5d1406fc4f5311365b9172a45253"} Oct 31 03:56:25.053051 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.053034 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" event={"ID":"dacc99d6-e9da-46e0-841d-14c5f54dd8fa","Type":"ContainerDied","Data":"f4b582bbb3a722098ff1cbe155a8985a4fcf0a438ecfedc4a5d01b6a1be298e7"} Oct 31 03:56:25.053340 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.052974 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5" Oct 31 03:56:25.064208 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.064169 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:25.070374 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.070349 5490 scope.go:122] "RemoveContainer" containerID="695ae0d222a4ce1f614904f66c785fb60a236e16e77f07fb2a54bde17bcc8031" Oct 31 03:56:25.070744 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:25.070717 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"695ae0d222a4ce1f614904f66c785fb60a236e16e77f07fb2a54bde17bcc8031\": container with ID starting with 695ae0d222a4ce1f614904f66c785fb60a236e16e77f07fb2a54bde17bcc8031 not found: ID does not exist" containerID="695ae0d222a4ce1f614904f66c785fb60a236e16e77f07fb2a54bde17bcc8031" Oct 31 03:56:25.070829 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.070756 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695ae0d222a4ce1f614904f66c785fb60a236e16e77f07fb2a54bde17bcc8031"} err="failed to get container status \"695ae0d222a4ce1f614904f66c785fb60a236e16e77f07fb2a54bde17bcc8031\": rpc error: code = NotFound desc = could not find container \"695ae0d222a4ce1f614904f66c785fb60a236e16e77f07fb2a54bde17bcc8031\": container with ID starting with 695ae0d222a4ce1f614904f66c785fb60a236e16e77f07fb2a54bde17bcc8031 not found: ID does not exist" Oct 31 03:56:25.070829 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.070781 5490 scope.go:122] "RemoveContainer" containerID="0998ecfcad5304891c2a3145d4d7a7e940ec5d1406fc4f5311365b9172a45253" Oct 31 03:56:25.086297 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.086259 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg"] Oct 31 03:56:25.087080 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.087061 5490 scope.go:122] "RemoveContainer" containerID="0998ecfcad5304891c2a3145d4d7a7e940ec5d1406fc4f5311365b9172a45253" Oct 31 03:56:25.087448 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:25.087419 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0998ecfcad5304891c2a3145d4d7a7e940ec5d1406fc4f5311365b9172a45253\": container with ID starting with 0998ecfcad5304891c2a3145d4d7a7e940ec5d1406fc4f5311365b9172a45253 not found: ID does not exist" containerID="0998ecfcad5304891c2a3145d4d7a7e940ec5d1406fc4f5311365b9172a45253" Oct 31 03:56:25.087515 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.087461 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0998ecfcad5304891c2a3145d4d7a7e940ec5d1406fc4f5311365b9172a45253"} err="failed to get container status \"0998ecfcad5304891c2a3145d4d7a7e940ec5d1406fc4f5311365b9172a45253\": rpc error: code = NotFound desc = could not find container \"0998ecfcad5304891c2a3145d4d7a7e940ec5d1406fc4f5311365b9172a45253\": container with ID starting with 0998ecfcad5304891c2a3145d4d7a7e940ec5d1406fc4f5311365b9172a45253 not found: ID does not exist" Oct 31 03:56:25.089960 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.089939 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-87gkg"] Oct 31 03:56:25.099939 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.099917 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5"] Oct 31 03:56:25.104306 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.104286 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b68780-openshift-default-579fc4bf79-2msk5"] Oct 31 03:56:25.246054 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.245945 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:25.586838 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.586751 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01e9fcdd-b435-47dc-a064-02ff1d074156" path="/var/lib/kubelet/pods/01e9fcdd-b435-47dc-a064-02ff1d074156/volumes" Oct 31 03:56:25.587306 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.587290 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="209cb838-5556-47a9-96ae-e2378ad8c8b5" path="/var/lib/kubelet/pods/209cb838-5556-47a9-96ae-e2378ad8c8b5/volumes" Oct 31 03:56:25.587897 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.587882 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baea2599-5723-4c5e-b063-e960ec09e121" path="/var/lib/kubelet/pods/baea2599-5723-4c5e-b063-e960ec09e121/volumes" Oct 31 03:56:25.588652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.588637 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dacc99d6-e9da-46e0-841d-14c5f54dd8fa" path="/var/lib/kubelet/pods/dacc99d6-e9da-46e0-841d-14c5f54dd8fa/volumes" Oct 31 03:56:25.997947 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:25.997785 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:26.882191 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:26.882121 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:27.605931 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.605898 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:56:27.617001 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.616969 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:56:27.665561 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.665521 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-workload-certs\") pod \"05bbcb81-55ce-4387-89c3-f6350fe8636f\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " Oct 31 03:56:27.665769 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.665626 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-workload-certs\") pod \"824618ba-ce21-4eeb-bf27-f1a861327863\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " Oct 31 03:56:27.666039 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.666005 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-workload-certs" pod "05bbcb81-55ce-4387-89c3-f6350fe8636f" (UID: "05bbcb81-55ce-4387-89c3-f6350fe8636f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.666138 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.666098 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-workload-certs" pod "824618ba-ce21-4eeb-bf27-f1a861327863" (UID: "824618ba-ce21-4eeb-bf27-f1a861327863"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.666138 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.666124 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-credential-socket\") pod \"824618ba-ce21-4eeb-bf27-f1a861327863\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " Oct 31 03:56:27.666238 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.666164 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05bbcb81-55ce-4387-89c3-f6350fe8636f-kube-api-access-jcs5h\" (UniqueName: \"kubernetes.io/projected/05bbcb81-55ce-4387-89c3-f6350fe8636f-kube-api-access-jcs5h\") pod \"05bbcb81-55ce-4387-89c3-f6350fe8636f\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " Oct 31 03:56:27.666238 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.666198 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824618ba-ce21-4eeb-bf27-f1a861327863-kube-api-access-fwkln\" (UniqueName: \"kubernetes.io/projected/824618ba-ce21-4eeb-bf27-f1a861327863-kube-api-access-fwkln\") pod \"824618ba-ce21-4eeb-bf27-f1a861327863\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " Oct 31 03:56:27.666385 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.666304 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-credential-socket\") pod \"05bbcb81-55ce-4387-89c3-f6350fe8636f\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " Oct 31 03:56:27.666385 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.666332 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-podinfo\") pod \"05bbcb81-55ce-4387-89c3-f6350fe8636f\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " Oct 31 03:56:27.666385 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.666364 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-token\" (UniqueName: \"kubernetes.io/projected/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-token\") pod \"05bbcb81-55ce-4387-89c3-f6350fe8636f\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " Oct 31 03:56:27.666559 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.666406 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-workload-socket\") pod \"824618ba-ce21-4eeb-bf27-f1a861327863\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " Oct 31 03:56:27.666559 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.666423 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-credential-socket" pod "824618ba-ce21-4eeb-bf27-f1a861327863" (UID: "824618ba-ce21-4eeb-bf27-f1a861327863"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.666559 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.666460 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05bbcb81-55ce-4387-89c3-f6350fe8636f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05bbcb81-55ce-4387-89c3-f6350fe8636f-istiod-ca-cert\") pod \"05bbcb81-55ce-4387-89c3-f6350fe8636f\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " Oct 31 03:56:27.666559 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.666502 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/824618ba-ce21-4eeb-bf27-f1a861327863-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824618ba-ce21-4eeb-bf27-f1a861327863-istiod-ca-cert\") pod \"824618ba-ce21-4eeb-bf27-f1a861327863\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " Oct 31 03:56:27.666559 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.666552 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-envoy\") pod \"05bbcb81-55ce-4387-89c3-f6350fe8636f\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " Oct 31 03:56:27.666932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.666597 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-data\") pod \"05bbcb81-55ce-4387-89c3-f6350fe8636f\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " Oct 31 03:56:27.666932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.666645 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824618ba-ce21-4eeb-bf27-f1a861327863-istio-token\" (UniqueName: \"kubernetes.io/projected/824618ba-ce21-4eeb-bf27-f1a861327863-istio-token\") pod \"824618ba-ce21-4eeb-bf27-f1a861327863\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " Oct 31 03:56:27.666932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.666661 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-credential-socket" pod "05bbcb81-55ce-4387-89c3-f6350fe8636f" (UID: "05bbcb81-55ce-4387-89c3-f6350fe8636f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.666932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.666698 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-istio-data\") pod \"824618ba-ce21-4eeb-bf27-f1a861327863\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " Oct 31 03:56:27.666932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.666737 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/824618ba-ce21-4eeb-bf27-f1a861327863-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824618ba-ce21-4eeb-bf27-f1a861327863-istio-podinfo\") pod \"824618ba-ce21-4eeb-bf27-f1a861327863\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " Oct 31 03:56:27.666932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.666804 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-istio-envoy\") pod \"824618ba-ce21-4eeb-bf27-f1a861327863\" (UID: \"824618ba-ce21-4eeb-bf27-f1a861327863\") " Oct 31 03:56:27.666932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.666891 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-workload-socket\") pod \"05bbcb81-55ce-4387-89c3-f6350fe8636f\" (UID: \"05bbcb81-55ce-4387-89c3-f6350fe8636f\") " Oct 31 03:56:27.667387 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.667362 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824618ba-ce21-4eeb-bf27-f1a861327863-istiod-ca-cert" pod "824618ba-ce21-4eeb-bf27-f1a861327863" (UID: "824618ba-ce21-4eeb-bf27-f1a861327863"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:27.667635 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.667534 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.667773 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.667639 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824618ba-ce21-4eeb-bf27-f1a861327863-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.667773 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.667649 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.667773 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.667658 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.667773 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.667667 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.668001 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.667906 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-workload-socket" pod "824618ba-ce21-4eeb-bf27-f1a861327863" (UID: "824618ba-ce21-4eeb-bf27-f1a861327863"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.668517 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.668242 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05bbcb81-55ce-4387-89c3-f6350fe8636f-istiod-ca-cert" pod "05bbcb81-55ce-4387-89c3-f6350fe8636f" (UID: "05bbcb81-55ce-4387-89c3-f6350fe8636f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:27.668621 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.668529 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-workload-socket" pod "05bbcb81-55ce-4387-89c3-f6350fe8636f" (UID: "05bbcb81-55ce-4387-89c3-f6350fe8636f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.669837 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.669802 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05bbcb81-55ce-4387-89c3-f6350fe8636f-kube-api-access-jcs5h" pod "05bbcb81-55ce-4387-89c3-f6350fe8636f" (UID: "05bbcb81-55ce-4387-89c3-f6350fe8636f"). InnerVolumeSpecName "kube-api-access-jcs5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:27.670142 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.670104 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-istio-data" pod "824618ba-ce21-4eeb-bf27-f1a861327863" (UID: "824618ba-ce21-4eeb-bf27-f1a861327863"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.670367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.670343 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-data" pod "05bbcb81-55ce-4387-89c3-f6350fe8636f" (UID: "05bbcb81-55ce-4387-89c3-f6350fe8636f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.671927 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.671898 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-istio-envoy" pod "824618ba-ce21-4eeb-bf27-f1a861327863" (UID: "824618ba-ce21-4eeb-bf27-f1a861327863"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.672271 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.672231 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824618ba-ce21-4eeb-bf27-f1a861327863-kube-api-access-fwkln" pod "824618ba-ce21-4eeb-bf27-f1a861327863" (UID: "824618ba-ce21-4eeb-bf27-f1a861327863"). InnerVolumeSpecName "kube-api-access-fwkln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:27.672271 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.672246 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-podinfo" pod "05bbcb81-55ce-4387-89c3-f6350fe8636f" (UID: "05bbcb81-55ce-4387-89c3-f6350fe8636f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:27.672895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.672409 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/824618ba-ce21-4eeb-bf27-f1a861327863-istio-podinfo" pod "824618ba-ce21-4eeb-bf27-f1a861327863" (UID: "824618ba-ce21-4eeb-bf27-f1a861327863"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:27.673307 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.673282 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824618ba-ce21-4eeb-bf27-f1a861327863-istio-token" pod "824618ba-ce21-4eeb-bf27-f1a861327863" (UID: "824618ba-ce21-4eeb-bf27-f1a861327863"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:27.674959 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.674937 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-token" pod "05bbcb81-55ce-4387-89c3-f6350fe8636f" (UID: "05bbcb81-55ce-4387-89c3-f6350fe8636f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:27.675051 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.674938 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-envoy" pod "05bbcb81-55ce-4387-89c3-f6350fe8636f" (UID: "05bbcb81-55ce-4387-89c3-f6350fe8636f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.694517 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.694460 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:56:27.768583 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.768540 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-podinfo\") pod \"8317b14b-20f5-4404-8f4c-bf315ec53736\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " Oct 31 03:56:27.768830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.768807 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-workload-socket\") pod \"8317b14b-20f5-4404-8f4c-bf315ec53736\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " Oct 31 03:56:27.768935 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.768889 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-workload-certs\") pod \"8317b14b-20f5-4404-8f4c-bf315ec53736\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " Oct 31 03:56:27.769010 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.768929 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-data\") pod \"8317b14b-20f5-4404-8f4c-bf315ec53736\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " Oct 31 03:56:27.769010 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.768991 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8317b14b-20f5-4404-8f4c-bf315ec53736-kube-api-access-w775l\" (UniqueName: \"kubernetes.io/projected/8317b14b-20f5-4404-8f4c-bf315ec53736-kube-api-access-w775l\") pod \"8317b14b-20f5-4404-8f4c-bf315ec53736\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " Oct 31 03:56:27.769130 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.769105 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-envoy\") pod \"8317b14b-20f5-4404-8f4c-bf315ec53736\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " Oct 31 03:56:27.769190 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.769134 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-credential-socket\") pod \"8317b14b-20f5-4404-8f4c-bf315ec53736\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " Oct 31 03:56:27.769190 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.769154 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-workload-socket" pod "8317b14b-20f5-4404-8f4c-bf315ec53736" (UID: "8317b14b-20f5-4404-8f4c-bf315ec53736"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.769307 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.769197 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-token\" (UniqueName: \"kubernetes.io/projected/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-token\") pod \"8317b14b-20f5-4404-8f4c-bf315ec53736\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " Oct 31 03:56:27.769307 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.769230 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8317b14b-20f5-4404-8f4c-bf315ec53736-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8317b14b-20f5-4404-8f4c-bf315ec53736-istiod-ca-cert\") pod \"8317b14b-20f5-4404-8f4c-bf315ec53736\" (UID: \"8317b14b-20f5-4404-8f4c-bf315ec53736\") " Oct 31 03:56:27.769421 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.769379 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-data" pod "8317b14b-20f5-4404-8f4c-bf315ec53736" (UID: "8317b14b-20f5-4404-8f4c-bf315ec53736"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.769756 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.769732 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8317b14b-20f5-4404-8f4c-bf315ec53736-istiod-ca-cert" pod "8317b14b-20f5-4404-8f4c-bf315ec53736" (UID: "8317b14b-20f5-4404-8f4c-bf315ec53736"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:27.769988 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.769965 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-workload-certs" pod "8317b14b-20f5-4404-8f4c-bf315ec53736" (UID: "8317b14b-20f5-4404-8f4c-bf315ec53736"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.770085 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.770039 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.770085 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.770063 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.770085 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.770077 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.770270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.770090 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05bbcb81-55ce-4387-89c3-f6350fe8636f-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.770270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.770104 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.770270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.770117 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.770270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.770130 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/824618ba-ce21-4eeb-bf27-f1a861327863-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.770270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.770143 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.770270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.770157 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.770270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.770163 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-credential-socket" pod "8317b14b-20f5-4404-8f4c-bf315ec53736" (UID: "8317b14b-20f5-4404-8f4c-bf315ec53736"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.770270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.770170 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824618ba-ce21-4eeb-bf27-f1a861327863-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.770270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.770203 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824618ba-ce21-4eeb-bf27-f1a861327863-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.770270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.770220 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.770270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.770236 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05bbcb81-55ce-4387-89c3-f6350fe8636f-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.770270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.770252 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcs5h\" (UniqueName: \"kubernetes.io/projected/05bbcb81-55ce-4387-89c3-f6350fe8636f-kube-api-access-jcs5h\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.770270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.770267 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwkln\" (UniqueName: \"kubernetes.io/projected/824618ba-ce21-4eeb-bf27-f1a861327863-kube-api-access-fwkln\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.770942 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.770282 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8317b14b-20f5-4404-8f4c-bf315ec53736-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.771447 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.771419 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-podinfo" pod "8317b14b-20f5-4404-8f4c-bf315ec53736" (UID: "8317b14b-20f5-4404-8f4c-bf315ec53736"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:27.772097 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.772064 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-envoy" pod "8317b14b-20f5-4404-8f4c-bf315ec53736" (UID: "8317b14b-20f5-4404-8f4c-bf315ec53736"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.772740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.772667 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-token" pod "8317b14b-20f5-4404-8f4c-bf315ec53736" (UID: "8317b14b-20f5-4404-8f4c-bf315ec53736"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:27.773167 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.773139 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8317b14b-20f5-4404-8f4c-bf315ec53736-kube-api-access-w775l" pod "8317b14b-20f5-4404-8f4c-bf315ec53736" (UID: "8317b14b-20f5-4404-8f4c-bf315ec53736"). InnerVolumeSpecName "kube-api-access-w775l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:27.776383 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.776361 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:56:27.871956 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.871871 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-envoy\") pod \"f57cc204-16b4-4735-82c1-7cee96c1704d\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " Oct 31 03:56:27.872145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.871955 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f57cc204-16b4-4735-82c1-7cee96c1704d-kube-api-access-shjhj\" (UniqueName: \"kubernetes.io/projected/f57cc204-16b4-4735-82c1-7cee96c1704d-kube-api-access-shjhj\") pod \"f57cc204-16b4-4735-82c1-7cee96c1704d\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " Oct 31 03:56:27.872145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.872044 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-workload-certs\") pod \"f57cc204-16b4-4735-82c1-7cee96c1704d\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " Oct 31 03:56:27.872145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.872092 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-workload-socket\") pod \"f57cc204-16b4-4735-82c1-7cee96c1704d\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " Oct 31 03:56:27.872145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.872123 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f57cc204-16b4-4735-82c1-7cee96c1704d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f57cc204-16b4-4735-82c1-7cee96c1704d-istiod-ca-cert\") pod \"f57cc204-16b4-4735-82c1-7cee96c1704d\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " Oct 31 03:56:27.872368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.872151 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-token\" (UniqueName: \"kubernetes.io/projected/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-token\") pod \"f57cc204-16b4-4735-82c1-7cee96c1704d\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " Oct 31 03:56:27.872368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.872251 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-data\") pod \"f57cc204-16b4-4735-82c1-7cee96c1704d\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " Oct 31 03:56:27.872368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.872285 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-credential-socket\") pod \"f57cc204-16b4-4735-82c1-7cee96c1704d\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " Oct 31 03:56:27.872368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.872336 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-podinfo\") pod \"f57cc204-16b4-4735-82c1-7cee96c1704d\" (UID: \"f57cc204-16b4-4735-82c1-7cee96c1704d\") " Oct 31 03:56:27.872591 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.872488 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-workload-socket" pod "f57cc204-16b4-4735-82c1-7cee96c1704d" (UID: "f57cc204-16b4-4735-82c1-7cee96c1704d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.872803 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.872775 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-data" pod "f57cc204-16b4-4735-82c1-7cee96c1704d" (UID: "f57cc204-16b4-4735-82c1-7cee96c1704d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.872912 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.872885 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-credential-socket" pod "f57cc204-16b4-4735-82c1-7cee96c1704d" (UID: "f57cc204-16b4-4735-82c1-7cee96c1704d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.872973 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.872951 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57cc204-16b4-4735-82c1-7cee96c1704d-istiod-ca-cert" pod "f57cc204-16b4-4735-82c1-7cee96c1704d" (UID: "f57cc204-16b4-4735-82c1-7cee96c1704d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:27.873230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.873213 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.873309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.873233 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.873309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.873248 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f57cc204-16b4-4735-82c1-7cee96c1704d-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.873309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.873262 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.873309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.873276 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.873309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.873290 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w775l\" (UniqueName: \"kubernetes.io/projected/8317b14b-20f5-4404-8f4c-bf315ec53736-kube-api-access-w775l\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.873309 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.873306 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.873651 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.873321 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8317b14b-20f5-4404-8f4c-bf315ec53736-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.873651 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.873335 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8317b14b-20f5-4404-8f4c-bf315ec53736-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.873651 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.873642 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-workload-certs" pod "f57cc204-16b4-4735-82c1-7cee96c1704d" (UID: "f57cc204-16b4-4735-82c1-7cee96c1704d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.874587 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.874554 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-envoy" pod "f57cc204-16b4-4735-82c1-7cee96c1704d" (UID: "f57cc204-16b4-4735-82c1-7cee96c1704d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:27.875251 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.875229 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-podinfo" pod "f57cc204-16b4-4735-82c1-7cee96c1704d" (UID: "f57cc204-16b4-4735-82c1-7cee96c1704d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:27.875827 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.875803 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57cc204-16b4-4735-82c1-7cee96c1704d-kube-api-access-shjhj" pod "f57cc204-16b4-4735-82c1-7cee96c1704d" (UID: "f57cc204-16b4-4735-82c1-7cee96c1704d"). InnerVolumeSpecName "kube-api-access-shjhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:27.876178 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.876139 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-token" pod "f57cc204-16b4-4735-82c1-7cee96c1704d" (UID: "f57cc204-16b4-4735-82c1-7cee96c1704d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:27.970983 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.970942 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:27.975461 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.975429 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.975461 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.975460 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shjhj\" (UniqueName: \"kubernetes.io/projected/f57cc204-16b4-4735-82c1-7cee96c1704d-kube-api-access-shjhj\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.975713 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.975475 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.975713 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.975489 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.975713 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.975503 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:27.975713 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:27.975519 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f57cc204-16b4-4735-82c1-7cee96c1704d-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:28.084290 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.084249 5490 generic.go:356] "Generic (PLEG): container finished" podID="8317b14b-20f5-4404-8f4c-bf315ec53736" containerID="75d33ed8265ccc3c5cd1979a992cbb481d894e7d6715fc035d67fb91b327dccf" exitCode=0 Oct 31 03:56:28.084503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.084306 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" Oct 31 03:56:28.084503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.084331 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" event={"ID":"8317b14b-20f5-4404-8f4c-bf315ec53736","Type":"ContainerDied","Data":"75d33ed8265ccc3c5cd1979a992cbb481d894e7d6715fc035d67fb91b327dccf"} Oct 31 03:56:28.084503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.084385 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h" event={"ID":"8317b14b-20f5-4404-8f4c-bf315ec53736","Type":"ContainerDied","Data":"928bb517591b623cdcc2ae0773c070758112ff881c842bc2f4a52c923959e550"} Oct 31 03:56:28.084503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.084420 5490 scope.go:122] "RemoveContainer" containerID="75d33ed8265ccc3c5cd1979a992cbb481d894e7d6715fc035d67fb91b327dccf" Oct 31 03:56:28.086038 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.086010 5490 generic.go:356] "Generic (PLEG): container finished" podID="824618ba-ce21-4eeb-bf27-f1a861327863" containerID="41f67ea9dd6fb51c2bb5ab18a755f9a86020168a384eca37e881b3afe0bceb47" exitCode=0 Oct 31 03:56:28.086139 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.086047 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" Oct 31 03:56:28.086139 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.086087 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" event={"ID":"824618ba-ce21-4eeb-bf27-f1a861327863","Type":"ContainerDied","Data":"41f67ea9dd6fb51c2bb5ab18a755f9a86020168a384eca37e881b3afe0bceb47"} Oct 31 03:56:28.086139 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.086135 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl" event={"ID":"824618ba-ce21-4eeb-bf27-f1a861327863","Type":"ContainerDied","Data":"4b56be18a650aa009333128d27cdf579fcced1a750d5b44f2f8ee98f969feefa"} Oct 31 03:56:28.087856 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.087823 5490 generic.go:356] "Generic (PLEG): container finished" podID="05bbcb81-55ce-4387-89c3-f6350fe8636f" containerID="ce4f1160f273d64b012eb62dba9a79a15defc4613a3b6e8efbfe985e6ec2d6d0" exitCode=0 Oct 31 03:56:28.087981 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.087866 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" Oct 31 03:56:28.088044 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.087976 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" event={"ID":"05bbcb81-55ce-4387-89c3-f6350fe8636f","Type":"ContainerDied","Data":"ce4f1160f273d64b012eb62dba9a79a15defc4613a3b6e8efbfe985e6ec2d6d0"} Oct 31 03:56:28.088044 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.088018 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd" event={"ID":"05bbcb81-55ce-4387-89c3-f6350fe8636f","Type":"ContainerDied","Data":"21a8842aaace157244dd9e7a44d2ba6b0f18415fe58d4829ceea2fb2fad0bd08"} Oct 31 03:56:28.089831 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.089804 5490 generic.go:356] "Generic (PLEG): container finished" podID="f57cc204-16b4-4735-82c1-7cee96c1704d" containerID="977556c538ae57240b5b254d28dbffae276a3566ad2bf1217ca04aa05d4bf3b0" exitCode=0 Oct 31 03:56:28.089964 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.089850 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" Oct 31 03:56:28.089964 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.089862 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" event={"ID":"f57cc204-16b4-4735-82c1-7cee96c1704d","Type":"ContainerDied","Data":"977556c538ae57240b5b254d28dbffae276a3566ad2bf1217ca04aa05d4bf3b0"} Oct 31 03:56:28.089964 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.089906 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh" event={"ID":"f57cc204-16b4-4735-82c1-7cee96c1704d","Type":"ContainerDied","Data":"7a99bdb12b9060c5cf02942ab341f715cc430a9f268d937a3e8c0ce4bb1af3ff"} Oct 31 03:56:28.098828 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.098805 5490 scope.go:122] "RemoveContainer" containerID="75d33ed8265ccc3c5cd1979a992cbb481d894e7d6715fc035d67fb91b327dccf" Oct 31 03:56:28.099146 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:28.099127 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d33ed8265ccc3c5cd1979a992cbb481d894e7d6715fc035d67fb91b327dccf\": container with ID starting with 75d33ed8265ccc3c5cd1979a992cbb481d894e7d6715fc035d67fb91b327dccf not found: ID does not exist" containerID="75d33ed8265ccc3c5cd1979a992cbb481d894e7d6715fc035d67fb91b327dccf" Oct 31 03:56:28.099225 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.099155 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d33ed8265ccc3c5cd1979a992cbb481d894e7d6715fc035d67fb91b327dccf"} err="failed to get container status \"75d33ed8265ccc3c5cd1979a992cbb481d894e7d6715fc035d67fb91b327dccf\": rpc error: code = NotFound desc = could not find container \"75d33ed8265ccc3c5cd1979a992cbb481d894e7d6715fc035d67fb91b327dccf\": container with ID starting with 75d33ed8265ccc3c5cd1979a992cbb481d894e7d6715fc035d67fb91b327dccf not found: ID does not exist" Oct 31 03:56:28.099225 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.099173 5490 scope.go:122] "RemoveContainer" containerID="41f67ea9dd6fb51c2bb5ab18a755f9a86020168a384eca37e881b3afe0bceb47" Oct 31 03:56:28.110827 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.110807 5490 scope.go:122] "RemoveContainer" containerID="41f67ea9dd6fb51c2bb5ab18a755f9a86020168a384eca37e881b3afe0bceb47" Oct 31 03:56:28.111144 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:28.111119 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f67ea9dd6fb51c2bb5ab18a755f9a86020168a384eca37e881b3afe0bceb47\": container with ID starting with 41f67ea9dd6fb51c2bb5ab18a755f9a86020168a384eca37e881b3afe0bceb47 not found: ID does not exist" containerID="41f67ea9dd6fb51c2bb5ab18a755f9a86020168a384eca37e881b3afe0bceb47" Oct 31 03:56:28.111234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.111152 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f67ea9dd6fb51c2bb5ab18a755f9a86020168a384eca37e881b3afe0bceb47"} err="failed to get container status \"41f67ea9dd6fb51c2bb5ab18a755f9a86020168a384eca37e881b3afe0bceb47\": rpc error: code = NotFound desc = could not find container \"41f67ea9dd6fb51c2bb5ab18a755f9a86020168a384eca37e881b3afe0bceb47\": container with ID starting with 41f67ea9dd6fb51c2bb5ab18a755f9a86020168a384eca37e881b3afe0bceb47 not found: ID does not exist" Oct 31 03:56:28.111234 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.111176 5490 scope.go:122] "RemoveContainer" containerID="ce4f1160f273d64b012eb62dba9a79a15defc4613a3b6e8efbfe985e6ec2d6d0" Oct 31 03:56:28.114433 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.114403 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:28.128829 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.128733 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd"] Oct 31 03:56:28.134357 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.134323 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8k5hd"] Oct 31 03:56:28.135990 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.135960 5490 scope.go:122] "RemoveContainer" containerID="ce4f1160f273d64b012eb62dba9a79a15defc4613a3b6e8efbfe985e6ec2d6d0" Oct 31 03:56:28.136333 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:28.136314 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce4f1160f273d64b012eb62dba9a79a15defc4613a3b6e8efbfe985e6ec2d6d0\": container with ID starting with ce4f1160f273d64b012eb62dba9a79a15defc4613a3b6e8efbfe985e6ec2d6d0 not found: ID does not exist" containerID="ce4f1160f273d64b012eb62dba9a79a15defc4613a3b6e8efbfe985e6ec2d6d0" Oct 31 03:56:28.136420 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.136342 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce4f1160f273d64b012eb62dba9a79a15defc4613a3b6e8efbfe985e6ec2d6d0"} err="failed to get container status \"ce4f1160f273d64b012eb62dba9a79a15defc4613a3b6e8efbfe985e6ec2d6d0\": rpc error: code = NotFound desc = could not find container \"ce4f1160f273d64b012eb62dba9a79a15defc4613a3b6e8efbfe985e6ec2d6d0\": container with ID starting with ce4f1160f273d64b012eb62dba9a79a15defc4613a3b6e8efbfe985e6ec2d6d0 not found: ID does not exist" Oct 31 03:56:28.136420 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.136360 5490 scope.go:122] "RemoveContainer" containerID="977556c538ae57240b5b254d28dbffae276a3566ad2bf1217ca04aa05d4bf3b0" Oct 31 03:56:28.139862 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.139124 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl"] Oct 31 03:56:28.144181 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.144142 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-9frxl"] Oct 31 03:56:28.149337 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.149318 5490 scope.go:122] "RemoveContainer" containerID="977556c538ae57240b5b254d28dbffae276a3566ad2bf1217ca04aa05d4bf3b0" Oct 31 03:56:28.149735 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:28.149702 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"977556c538ae57240b5b254d28dbffae276a3566ad2bf1217ca04aa05d4bf3b0\": container with ID starting with 977556c538ae57240b5b254d28dbffae276a3566ad2bf1217ca04aa05d4bf3b0 not found: ID does not exist" containerID="977556c538ae57240b5b254d28dbffae276a3566ad2bf1217ca04aa05d4bf3b0" Oct 31 03:56:28.150558 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.150515 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"977556c538ae57240b5b254d28dbffae276a3566ad2bf1217ca04aa05d4bf3b0"} err="failed to get container status \"977556c538ae57240b5b254d28dbffae276a3566ad2bf1217ca04aa05d4bf3b0\": rpc error: code = NotFound desc = could not find container \"977556c538ae57240b5b254d28dbffae276a3566ad2bf1217ca04aa05d4bf3b0\": container with ID starting with 977556c538ae57240b5b254d28dbffae276a3566ad2bf1217ca04aa05d4bf3b0 not found: ID does not exist" Oct 31 03:56:28.154350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.154329 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh"] Oct 31 03:56:28.163937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.163912 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-t4sjh"] Oct 31 03:56:28.176408 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.176377 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h"] Oct 31 03:56:28.180229 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.180185 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-4hc7h"] Oct 31 03:56:28.651726 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.651669 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59"] Oct 31 03:56:28.652478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652438 5490 cpu_manager.go:414] "containerMap: removing container" podUID="209cb838-5556-47a9-96ae-e2378ad8c8b5" containerName="istio-proxy" Oct 31 03:56:28.652478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652461 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="209cb838-5556-47a9-96ae-e2378ad8c8b5" containerName="istio-proxy" Oct 31 03:56:28.652478 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652481 5490 cpu_manager.go:414] "containerMap: removing container" podUID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" containerName="istio-proxy" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652488 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" containerName="istio-proxy" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652498 5490 cpu_manager.go:414] "containerMap: removing container" podUID="824618ba-ce21-4eeb-bf27-f1a861327863" containerName="istio-proxy" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652506 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824618ba-ce21-4eeb-bf27-f1a861327863" containerName="istio-proxy" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652528 5490 cpu_manager.go:414] "containerMap: removing container" podUID="dacc99d6-e9da-46e0-841d-14c5f54dd8fa" containerName="istio-proxy" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652536 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dacc99d6-e9da-46e0-841d-14c5f54dd8fa" containerName="istio-proxy" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652550 5490 cpu_manager.go:414] "containerMap: removing container" podUID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" containerName="istio-proxy" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652556 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" containerName="istio-proxy" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652568 5490 cpu_manager.go:414] "containerMap: removing container" podUID="8317b14b-20f5-4404-8f4c-bf315ec53736" containerName="istio-proxy" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652574 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8317b14b-20f5-4404-8f4c-bf315ec53736" containerName="istio-proxy" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652595 5490 cpu_manager.go:414] "containerMap: removing container" podUID="1ce35023-5f08-4356-8e09-4328952c6040" containerName="istio-proxy" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652601 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce35023-5f08-4356-8e09-4328952c6040" containerName="istio-proxy" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652615 5490 cpu_manager.go:414] "containerMap: removing container" podUID="604d3706-e3e3-4fe2-a9bf-5472a41ee2f0" containerName="maas-api" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652621 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604d3706-e3e3-4fe2-a9bf-5472a41ee2f0" containerName="maas-api" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652632 5490 cpu_manager.go:414] "containerMap: removing container" podUID="baea2599-5723-4c5e-b063-e960ec09e121" containerName="istio-proxy" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652640 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baea2599-5723-4c5e-b063-e960ec09e121" containerName="istio-proxy" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652688 5490 cpu_manager.go:414] "containerMap: removing container" podUID="b7c0d839-b3a4-463b-a3a8-3733609d4b89" containerName="maas-api" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652698 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c0d839-b3a4-463b-a3a8-3733609d4b89" containerName="maas-api" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652719 5490 cpu_manager.go:414] "containerMap: removing container" podUID="cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" containerName="istio-proxy" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652726 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" containerName="istio-proxy" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652736 5490 cpu_manager.go:414] "containerMap: removing container" podUID="01e9fcdd-b435-47dc-a064-02ff1d074156" containerName="istio-proxy" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652743 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e9fcdd-b435-47dc-a064-02ff1d074156" containerName="istio-proxy" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652760 5490 cpu_manager.go:414] "containerMap: removing container" podUID="b8353165-d9b4-445a-9a11-5fdf04307860" containerName="maas-api" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652768 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8353165-d9b4-445a-9a11-5fdf04307860" containerName="maas-api" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652781 5490 cpu_manager.go:414] "containerMap: removing container" podUID="da913fbf-0ba8-4607-be4c-5197a22d1a7d" containerName="maas-api" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652788 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da913fbf-0ba8-4607-be4c-5197a22d1a7d" containerName="maas-api" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652800 5490 cpu_manager.go:414] "containerMap: removing container" podUID="253719d9-0fac-4d41-bfc3-d4d359ba04fe" containerName="maas-api" Oct 31 03:56:28.652801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652808 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="253719d9-0fac-4d41-bfc3-d4d359ba04fe" containerName="maas-api" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652824 5490 cpu_manager.go:414] "containerMap: removing container" podUID="f57cc204-16b4-4735-82c1-7cee96c1704d" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652831 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f57cc204-16b4-4735-82c1-7cee96c1704d" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652841 5490 cpu_manager.go:414] "containerMap: removing container" podUID="05bbcb81-55ce-4387-89c3-f6350fe8636f" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652847 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05bbcb81-55ce-4387-89c3-f6350fe8636f" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652861 5490 cpu_manager.go:414] "containerMap: removing container" podUID="bae51678-b17d-4a9c-98da-77d3ecf0d6ed" containerName="payload-pre-processing" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652868 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae51678-b17d-4a9c-98da-77d3ecf0d6ed" containerName="payload-pre-processing" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652890 5490 cpu_manager.go:414] "containerMap: removing container" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652897 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652910 5490 cpu_manager.go:414] "containerMap: removing container" podUID="015afcb4-d156-4163-a342-f84471c9dd51" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652918 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="015afcb4-d156-4163-a342-f84471c9dd51" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652932 5490 cpu_manager.go:414] "containerMap: removing container" podUID="015afcb4-d156-4163-a342-f84471c9dd51" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.652939 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="015afcb4-d156-4163-a342-f84471c9dd51" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653113 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c0d839-b3a4-463b-a3a8-3733609d4b89" containerName="maas-api" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653128 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3630f1-9fe3-4a38-8095-2dba83b4e1a7" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653141 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e9fcdd-b435-47dc-a064-02ff1d074156" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653151 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8353165-d9b4-445a-9a11-5fdf04307860" containerName="maas-api" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653159 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="da913fbf-0ba8-4607-be4c-5197a22d1a7d" containerName="maas-api" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653167 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="253719d9-0fac-4d41-bfc3-d4d359ba04fe" containerName="maas-api" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653177 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="f57cc204-16b4-4735-82c1-7cee96c1704d" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653185 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="05bbcb81-55ce-4387-89c3-f6350fe8636f" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653197 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae51678-b17d-4a9c-98da-77d3ecf0d6ed" containerName="payload-pre-processing" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653210 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="7880b1c4-cc27-4692-ae95-5682724c2f4c" containerName="main" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653220 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="015afcb4-d156-4163-a342-f84471c9dd51" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653229 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="015afcb4-d156-4163-a342-f84471c9dd51" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653240 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce35023-5f08-4356-8e09-4328952c6040" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653250 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="209cb838-5556-47a9-96ae-e2378ad8c8b5" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653261 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653268 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="824618ba-ce21-4eeb-bf27-f1a861327863" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653278 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="dacc99d6-e9da-46e0-841d-14c5f54dd8fa" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653288 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="8317b14b-20f5-4404-8f4c-bf315ec53736" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653300 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce35023-5f08-4356-8e09-4328952c6040" containerName="istio-proxy" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653310 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="604d3706-e3e3-4fe2-a9bf-5472a41ee2f0" containerName="maas-api" Oct 31 03:56:28.654198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.653318 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="baea2599-5723-4c5e-b063-e960ec09e121" containerName="istio-proxy" Oct 31 03:56:28.659188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.659166 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.672352 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.672312 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-d04aad-openshift-default-dockercfg-ljdjv\"" Oct 31 03:56:28.674240 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.674212 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59"] Oct 31 03:56:28.787210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.787173 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istiod-ca-cert\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.787210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.787212 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-envoy\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.787410 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.787236 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-credential-socket\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.787410 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.787333 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-data\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.787504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.787449 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-podinfo\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.787504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.787479 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-workload-socket\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.787581 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.787559 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-workload-certs\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.787618 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.787590 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-token\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.787755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.787741 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4k2g\" (UniqueName: \"kubernetes.io/projected/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-kube-api-access-p4k2g\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.836499 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.836437 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:28.889272 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.889230 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-podinfo\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.889272 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.889269 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-workload-socket\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.889540 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.889293 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-workload-certs\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.889540 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.889412 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-token\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.889656 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.889558 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4k2g\" (UniqueName: \"kubernetes.io/projected/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-kube-api-access-p4k2g\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.889755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.889711 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istiod-ca-cert\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.889755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.889721 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-workload-certs\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.889755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.889736 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-workload-socket\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.889755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.889744 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-envoy\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.889970 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.889788 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-credential-socket\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.889970 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.889821 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-data\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.890192 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.890153 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-data\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.890404 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.890385 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-credential-socket\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.890472 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.890431 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istiod-ca-cert\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.892057 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.892031 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-envoy\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.892190 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.892082 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-podinfo\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.898515 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.898488 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-token\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.898980 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.898959 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4k2g\" (UniqueName: \"kubernetes.io/projected/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-kube-api-access-p4k2g\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:28.975083 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:28.975036 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:29.128305 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.128245 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:29.307162 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.307124 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59"] Oct 31 03:56:29.311614 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.310800 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:56:29.311614 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.310900 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:56:29.311614 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.310948 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:56:29.316250 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.316215 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:29.422553 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.422509 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" podUID="ceb68ad8-3da3-4469-9207-cb473d1dad9e" containerName="payload-processing" containerID="cri-o://126faaaf3ca3b600d22cb5fb1f588765b8b090637cd52a9d97e31ae85fec311e" gracePeriod=25 Oct 31 03:56:29.450542 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.450482 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" podUID="5d9ae28c-045f-456f-bbd7-8105b6e0077d" containerName="payload-pre-processing" containerID="cri-o://ea9dab4883d0e36f5ffd760db61792fa0b387dc6d8e994e50f2e46744896c43b" gracePeriod=25 Oct 31 03:56:29.588893 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.588825 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05bbcb81-55ce-4387-89c3-f6350fe8636f" path="/var/lib/kubelet/pods/05bbcb81-55ce-4387-89c3-f6350fe8636f/volumes" Oct 31 03:56:29.589885 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.589839 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824618ba-ce21-4eeb-bf27-f1a861327863" path="/var/lib/kubelet/pods/824618ba-ce21-4eeb-bf27-f1a861327863/volumes" Oct 31 03:56:29.590704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.590670 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8317b14b-20f5-4404-8f4c-bf315ec53736" path="/var/lib/kubelet/pods/8317b14b-20f5-4404-8f4c-bf315ec53736/volumes" Oct 31 03:56:29.591866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.591845 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f57cc204-16b4-4735-82c1-7cee96c1704d" path="/var/lib/kubelet/pods/f57cc204-16b4-4735-82c1-7cee96c1704d/volumes" Oct 31 03:56:29.764145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.764116 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" Oct 31 03:56:29.802032 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.802000 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" Oct 31 03:56:29.921831 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.921796 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceb68ad8-3da3-4469-9207-cb473d1dad9e-kube-api-access-fg4xp\" (UniqueName: \"kubernetes.io/projected/ceb68ad8-3da3-4469-9207-cb473d1dad9e-kube-api-access-fg4xp\") pod \"ceb68ad8-3da3-4469-9207-cb473d1dad9e\" (UID: \"ceb68ad8-3da3-4469-9207-cb473d1dad9e\") " Oct 31 03:56:29.922056 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.921888 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9ae28c-045f-456f-bbd7-8105b6e0077d-kube-api-access-t6x2r\" (UniqueName: \"kubernetes.io/projected/5d9ae28c-045f-456f-bbd7-8105b6e0077d-kube-api-access-t6x2r\") pod \"5d9ae28c-045f-456f-bbd7-8105b6e0077d\" (UID: \"5d9ae28c-045f-456f-bbd7-8105b6e0077d\") " Oct 31 03:56:29.922056 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.921951 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceb68ad8-3da3-4469-9207-cb473d1dad9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb68ad8-3da3-4469-9207-cb473d1dad9e-plugins-config-volume\") pod \"ceb68ad8-3da3-4469-9207-cb473d1dad9e\" (UID: \"ceb68ad8-3da3-4469-9207-cb473d1dad9e\") " Oct 31 03:56:29.922192 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.922053 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d9ae28c-045f-456f-bbd7-8105b6e0077d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9ae28c-045f-456f-bbd7-8105b6e0077d-plugins-config-volume\") pod \"5d9ae28c-045f-456f-bbd7-8105b6e0077d\" (UID: \"5d9ae28c-045f-456f-bbd7-8105b6e0077d\") " Oct 31 03:56:29.922386 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.922353 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb68ad8-3da3-4469-9207-cb473d1dad9e-plugins-config-volume" pod "ceb68ad8-3da3-4469-9207-cb473d1dad9e" (UID: "ceb68ad8-3da3-4469-9207-cb473d1dad9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:29.922509 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.922419 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9ae28c-045f-456f-bbd7-8105b6e0077d-plugins-config-volume" pod "5d9ae28c-045f-456f-bbd7-8105b6e0077d" (UID: "5d9ae28c-045f-456f-bbd7-8105b6e0077d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:29.922628 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.922609 5490 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d9ae28c-045f-456f-bbd7-8105b6e0077d-plugins-config-volume\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:29.922722 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.922631 5490 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ceb68ad8-3da3-4469-9207-cb473d1dad9e-plugins-config-volume\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:29.924191 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.924174 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb68ad8-3da3-4469-9207-cb473d1dad9e-kube-api-access-fg4xp" pod "ceb68ad8-3da3-4469-9207-cb473d1dad9e" (UID: "ceb68ad8-3da3-4469-9207-cb473d1dad9e"). InnerVolumeSpecName "kube-api-access-fg4xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:29.924249 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:29.924201 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9ae28c-045f-456f-bbd7-8105b6e0077d-kube-api-access-t6x2r" pod "5d9ae28c-045f-456f-bbd7-8105b6e0077d" (UID: "5d9ae28c-045f-456f-bbd7-8105b6e0077d"). InnerVolumeSpecName "kube-api-access-t6x2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:30.024362 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.024326 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg4xp\" (UniqueName: \"kubernetes.io/projected/ceb68ad8-3da3-4469-9207-cb473d1dad9e-kube-api-access-fg4xp\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:30.024362 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.024362 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6x2r\" (UniqueName: \"kubernetes.io/projected/5d9ae28c-045f-456f-bbd7-8105b6e0077d-kube-api-access-t6x2r\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:30.135532 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.134436 5490 generic.go:356] "Generic (PLEG): container finished" podID="5d9ae28c-045f-456f-bbd7-8105b6e0077d" containerID="ea9dab4883d0e36f5ffd760db61792fa0b387dc6d8e994e50f2e46744896c43b" exitCode=0 Oct 31 03:56:30.135532 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.134653 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" event={"ID":"5d9ae28c-045f-456f-bbd7-8105b6e0077d","Type":"ContainerDied","Data":"ea9dab4883d0e36f5ffd760db61792fa0b387dc6d8e994e50f2e46744896c43b"} Oct 31 03:56:30.135532 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.134750 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" event={"ID":"5d9ae28c-045f-456f-bbd7-8105b6e0077d","Type":"ContainerDied","Data":"0852fdfbf1e5f6cd30fcb20ab22db9d956ac7e32a3763ef4c2cf00476af6d4ae"} Oct 31 03:56:30.135532 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.134783 5490 scope.go:122] "RemoveContainer" containerID="ea9dab4883d0e36f5ffd760db61792fa0b387dc6d8e994e50f2e46744896c43b" Oct 31 03:56:30.135532 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.134892 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv" Oct 31 03:56:30.156762 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.156695 5490 generic.go:356] "Generic (PLEG): container finished" podID="ceb68ad8-3da3-4469-9207-cb473d1dad9e" containerID="126faaaf3ca3b600d22cb5fb1f588765b8b090637cd52a9d97e31ae85fec311e" exitCode=0 Oct 31 03:56:30.156911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.156789 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" event={"ID":"ceb68ad8-3da3-4469-9207-cb473d1dad9e","Type":"ContainerDied","Data":"126faaaf3ca3b600d22cb5fb1f588765b8b090637cd52a9d97e31ae85fec311e"} Oct 31 03:56:30.156911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.156835 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" event={"ID":"ceb68ad8-3da3-4469-9207-cb473d1dad9e","Type":"ContainerDied","Data":"dde47236d35d30542b18cc50e65a8a674ff740006ff0b861d0efb92f4d06621f"} Oct 31 03:56:30.156911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.156903 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2" Oct 31 03:56:30.173912 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.173817 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" event={"ID":"a1e68997-6d71-4e7e-a80c-ca5c29aa225e","Type":"ContainerStarted","Data":"e8d475349f1267087117267a60ec79eb175ce4d740bfa1eb2c99d2639f02c72f"} Oct 31 03:56:30.174190 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.174148 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" event={"ID":"a1e68997-6d71-4e7e-a80c-ca5c29aa225e","Type":"ContainerStarted","Data":"f9b221f94fed6c50e9d9c0e7c008058542e90e7dc00169ba5f99e165cc26c55b"} Oct 31 03:56:30.193042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.193010 5490 scope.go:122] "RemoveContainer" containerID="ea9dab4883d0e36f5ffd760db61792fa0b387dc6d8e994e50f2e46744896c43b" Oct 31 03:56:30.197043 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:30.196991 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea9dab4883d0e36f5ffd760db61792fa0b387dc6d8e994e50f2e46744896c43b\": container with ID starting with ea9dab4883d0e36f5ffd760db61792fa0b387dc6d8e994e50f2e46744896c43b not found: ID does not exist" containerID="ea9dab4883d0e36f5ffd760db61792fa0b387dc6d8e994e50f2e46744896c43b" Oct 31 03:56:30.197215 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.197044 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9dab4883d0e36f5ffd760db61792fa0b387dc6d8e994e50f2e46744896c43b"} err="failed to get container status \"ea9dab4883d0e36f5ffd760db61792fa0b387dc6d8e994e50f2e46744896c43b\": rpc error: code = NotFound desc = could not find container \"ea9dab4883d0e36f5ffd760db61792fa0b387dc6d8e994e50f2e46744896c43b\": container with ID starting with ea9dab4883d0e36f5ffd760db61792fa0b387dc6d8e994e50f2e46744896c43b not found: ID does not exist" Oct 31 03:56:30.197215 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.197071 5490 scope.go:122] "RemoveContainer" containerID="126faaaf3ca3b600d22cb5fb1f588765b8b090637cd52a9d97e31ae85fec311e" Oct 31 03:56:30.201341 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.201305 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv"] Oct 31 03:56:30.204020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.203985 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d86a77-69db594f6c-qmbxv"] Oct 31 03:56:30.210804 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.210784 5490 scope.go:122] "RemoveContainer" containerID="126faaaf3ca3b600d22cb5fb1f588765b8b090637cd52a9d97e31ae85fec311e" Oct 31 03:56:30.211127 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:30.211102 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"126faaaf3ca3b600d22cb5fb1f588765b8b090637cd52a9d97e31ae85fec311e\": container with ID starting with 126faaaf3ca3b600d22cb5fb1f588765b8b090637cd52a9d97e31ae85fec311e not found: ID does not exist" containerID="126faaaf3ca3b600d22cb5fb1f588765b8b090637cd52a9d97e31ae85fec311e" Oct 31 03:56:30.211194 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.211133 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126faaaf3ca3b600d22cb5fb1f588765b8b090637cd52a9d97e31ae85fec311e"} err="failed to get container status \"126faaaf3ca3b600d22cb5fb1f588765b8b090637cd52a9d97e31ae85fec311e\": rpc error: code = NotFound desc = could not find container \"126faaaf3ca3b600d22cb5fb1f588765b8b090637cd52a9d97e31ae85fec311e\": container with ID starting with 126faaaf3ca3b600d22cb5fb1f588765b8b090637cd52a9d97e31ae85fec311e not found: ID does not exist" Oct 31 03:56:30.248040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.247979 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:30.250010 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.249954 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" podStartSLOduration=2.249932826 podStartE2EDuration="2.249932826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:30.222069848 +0000 UTC m=+2424.756764802" watchObservedRunningTime="2026-10-31 03:56:30.249932826 +0000 UTC m=+2424.784627782" Oct 31 03:56:30.264879 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.264847 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2"] Oct 31 03:56:30.268641 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.268615 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d86a77-6589c944f7-w6hl2"] Oct 31 03:56:30.599587 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.599320 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:30.754628 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.754586 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:30.975415 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.975378 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:30.976876 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.976845 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 03:56:30.977019 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:30.976899 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" podUID="a1e68997-6d71-4e7e-a80c-ca5c29aa225e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 03:56:31.243620 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:31.243525 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:31.577188 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:31.577102 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d9ae28c-045f-456f-bbd7-8105b6e0077d" path="/var/lib/kubelet/pods/5d9ae28c-045f-456f-bbd7-8105b6e0077d/volumes" Oct 31 03:56:31.577688 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:31.577654 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceb68ad8-3da3-4469-9207-cb473d1dad9e" path="/var/lib/kubelet/pods/ceb68ad8-3da3-4469-9207-cb473d1dad9e/volumes" Oct 31 03:56:31.976492 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:31.976450 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 03:56:31.976924 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:31.976511 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" podUID="a1e68997-6d71-4e7e-a80c-ca5c29aa225e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 03:56:32.101856 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:32.101814 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:32.259502 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:32.259422 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:32.570342 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:32.570258 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:32.870953 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:32.870854 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:32.976359 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:32.976320 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 03:56:32.976543 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:32.976379 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" podUID="a1e68997-6d71-4e7e-a80c-ca5c29aa225e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 03:56:33.010568 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:33.010517 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:33.499982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:33.499916 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:33.651943 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:33.648369 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:33.950698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:33.949317 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:33.976320 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:33.976278 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 03:56:33.976502 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:33.976366 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" podUID="a1e68997-6d71-4e7e-a80c-ca5c29aa225e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 03:56:34.052704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.050861 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" containerID="cri-o://f91006e5ba6533082e0b81908d537ffce86d3551192c5cf45e9bbaf2299a8046" gracePeriod=45 Oct 31 03:56:34.052704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.051033 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" containerID="cri-o://47406f4963632d15d7744157303fb408aa472a0a64c5fcf2cfdc75f66054f241" gracePeriod=45 Oct 31 03:56:34.214066 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.213963 5490 generic.go:356] "Generic (PLEG): container finished" podID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerID="47406f4963632d15d7744157303fb408aa472a0a64c5fcf2cfdc75f66054f241" exitCode=0 Oct 31 03:56:34.214066 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.214060 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" event={"ID":"3f89ed71-a623-40af-a5b1-5f6333235c14","Type":"ContainerDied","Data":"47406f4963632d15d7744157303fb408aa472a0a64c5fcf2cfdc75f66054f241"} Oct 31 03:56:34.216141 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.216106 5490 generic.go:356] "Generic (PLEG): container finished" podID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerID="f91006e5ba6533082e0b81908d537ffce86d3551192c5cf45e9bbaf2299a8046" exitCode=0 Oct 31 03:56:34.216250 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.216242 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" event={"ID":"d61881b1-0191-43d7-9ac2-c060af4e8f6c","Type":"ContainerDied","Data":"f91006e5ba6533082e0b81908d537ffce86d3551192c5cf45e9bbaf2299a8046"} Oct 31 03:56:34.410941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.410905 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:56:34.422807 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.422771 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-home\" (UniqueName: \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-home\") pod \"3f89ed71-a623-40af-a5b1-5f6333235c14\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " Oct 31 03:56:34.422978 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.422948 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-tmp-dir\") pod \"3f89ed71-a623-40af-a5b1-5f6333235c14\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " Oct 31 03:56:34.423045 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.423011 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-home" pod "3f89ed71-a623-40af-a5b1-5f6333235c14" (UID: "3f89ed71-a623-40af-a5b1-5f6333235c14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:34.423102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.423079 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-model-cache\") pod \"3f89ed71-a623-40af-a5b1-5f6333235c14\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " Oct 31 03:56:34.423216 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.423199 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-dshm\") pod \"3f89ed71-a623-40af-a5b1-5f6333235c14\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " Oct 31 03:56:34.423294 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.423272 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-model-cache" pod "3f89ed71-a623-40af-a5b1-5f6333235c14" (UID: "3f89ed71-a623-40af-a5b1-5f6333235c14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:34.423351 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.423321 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f89ed71-a623-40af-a5b1-5f6333235c14-kube-api-access-xccbf\" (UniqueName: \"kubernetes.io/projected/3f89ed71-a623-40af-a5b1-5f6333235c14-kube-api-access-xccbf\") pod \"3f89ed71-a623-40af-a5b1-5f6333235c14\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " Oct 31 03:56:34.423503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.423469 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-tmp-dir" pod "3f89ed71-a623-40af-a5b1-5f6333235c14" (UID: "3f89ed71-a623-40af-a5b1-5f6333235c14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:34.423886 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.423864 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f89ed71-a623-40af-a5b1-5f6333235c14-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f89ed71-a623-40af-a5b1-5f6333235c14-tls-certs\") pod \"3f89ed71-a623-40af-a5b1-5f6333235c14\" (UID: \"3f89ed71-a623-40af-a5b1-5f6333235c14\") " Oct 31 03:56:34.424780 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.424746 5490 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-home\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:34.424878 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.424784 5490 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-tmp-dir\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:34.424878 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.424798 5490 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-model-cache\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:34.425579 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.425554 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f89ed71-a623-40af-a5b1-5f6333235c14-kube-api-access-xccbf" pod "3f89ed71-a623-40af-a5b1-5f6333235c14" (UID: "3f89ed71-a623-40af-a5b1-5f6333235c14"). InnerVolumeSpecName "kube-api-access-xccbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:34.467834 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.428854 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f89ed71-a623-40af-a5b1-5f6333235c14-tls-certs" pod "3f89ed71-a623-40af-a5b1-5f6333235c14" (UID: "3f89ed71-a623-40af-a5b1-5f6333235c14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:34.467834 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.434829 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-dshm" pod "3f89ed71-a623-40af-a5b1-5f6333235c14" (UID: "3f89ed71-a623-40af-a5b1-5f6333235c14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:34.467834 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.452894 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:56:34.525439 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.525403 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-home\" (UniqueName: \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-home\") pod \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " Oct 31 03:56:34.525630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.525474 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-tmp-dir\") pod \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " Oct 31 03:56:34.525630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.525555 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-model-cache\") pod \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " Oct 31 03:56:34.556704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.544073 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61881b1-0191-43d7-9ac2-c060af4e8f6c-kube-api-access-x6psd\" (UniqueName: \"kubernetes.io/projected/d61881b1-0191-43d7-9ac2-c060af4e8f6c-kube-api-access-x6psd\") pod \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " Oct 31 03:56:34.556704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.544344 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d61881b1-0191-43d7-9ac2-c060af4e8f6c-tls-certs\" (UniqueName: \"kubernetes.io/secret/d61881b1-0191-43d7-9ac2-c060af4e8f6c-tls-certs\") pod \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " Oct 31 03:56:34.556704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.544443 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-dshm\") pod \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\" (UID: \"d61881b1-0191-43d7-9ac2-c060af4e8f6c\") " Oct 31 03:56:34.556704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.545301 5490 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f89ed71-a623-40af-a5b1-5f6333235c14-dshm\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:34.556704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.545330 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xccbf\" (UniqueName: \"kubernetes.io/projected/3f89ed71-a623-40af-a5b1-5f6333235c14-kube-api-access-xccbf\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:34.556704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.545347 5490 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f89ed71-a623-40af-a5b1-5f6333235c14-tls-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:34.556704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.549774 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-home" pod "d61881b1-0191-43d7-9ac2-c060af4e8f6c" (UID: "d61881b1-0191-43d7-9ac2-c060af4e8f6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:34.556704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.552839 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-tmp-dir" pod "d61881b1-0191-43d7-9ac2-c060af4e8f6c" (UID: "d61881b1-0191-43d7-9ac2-c060af4e8f6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:34.556704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.552927 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-model-cache" pod "d61881b1-0191-43d7-9ac2-c060af4e8f6c" (UID: "d61881b1-0191-43d7-9ac2-c060af4e8f6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:34.559196 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.559154 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61881b1-0191-43d7-9ac2-c060af4e8f6c-kube-api-access-x6psd" pod "d61881b1-0191-43d7-9ac2-c060af4e8f6c" (UID: "d61881b1-0191-43d7-9ac2-c060af4e8f6c"). InnerVolumeSpecName "kube-api-access-x6psd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:34.559734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.559660 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d61881b1-0191-43d7-9ac2-c060af4e8f6c-tls-certs" pod "d61881b1-0191-43d7-9ac2-c060af4e8f6c" (UID: "d61881b1-0191-43d7-9ac2-c060af4e8f6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:34.560379 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.560354 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-dshm" pod "d61881b1-0191-43d7-9ac2-c060af4e8f6c" (UID: "d61881b1-0191-43d7-9ac2-c060af4e8f6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:34.652472 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.652401 5490 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d61881b1-0191-43d7-9ac2-c060af4e8f6c-tls-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:34.652472 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.652432 5490 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-dshm\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:34.652472 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.652447 5490 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-home\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:34.652472 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.652460 5490 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-tmp-dir\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:34.652472 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.652473 5490 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d61881b1-0191-43d7-9ac2-c060af4e8f6c-model-cache\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:34.652985 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.652496 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6psd\" (UniqueName: \"kubernetes.io/projected/d61881b1-0191-43d7-9ac2-c060af4e8f6c-kube-api-access-x6psd\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:34.747035 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.746832 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:34.976552 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.976510 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 03:56:34.976754 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:34.976580 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" podUID="a1e68997-6d71-4e7e-a80c-ca5c29aa225e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 03:56:35.225439 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:35.225394 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" Oct 31 03:56:35.225439 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:35.225401 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj" event={"ID":"3f89ed71-a623-40af-a5b1-5f6333235c14","Type":"ContainerDied","Data":"e89ecef5458145ff96ece614a5e3d3a2c8d1bb1471850d64b193b3ee7cb881fe"} Oct 31 03:56:35.246082 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:35.225480 5490 scope.go:122] "RemoveContainer" containerID="47406f4963632d15d7744157303fb408aa472a0a64c5fcf2cfdc75f66054f241" Oct 31 03:56:35.249230 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:35.249200 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" event={"ID":"d61881b1-0191-43d7-9ac2-c060af4e8f6c","Type":"ContainerDied","Data":"1c31c8b11b46b4b62fac912133ffabe125c629f1efa9040cc5c0482adb4f9659"} Oct 31 03:56:35.249498 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:35.249477 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6" Oct 31 03:56:35.296041 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:35.296006 5490 scope.go:122] "RemoveContainer" containerID="f91006e5ba6533082e0b81908d537ffce86d3551192c5cf45e9bbaf2299a8046" Oct 31 03:56:35.302971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:35.302941 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6"] Oct 31 03:56:35.308526 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:35.308497 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/distinct-w0-d86a77-kserve-85fb976dfd-xfxl6"] Oct 31 03:56:35.315304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:35.315282 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj"] Oct 31 03:56:35.318747 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:35.318721 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/distinct-2-w0-d86a77-kserve-5c987f98b9-bcjhj"] Oct 31 03:56:35.353873 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:35.352512 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:35.591018 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:35.590858 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:35.604507 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:35.604411 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" path="/var/lib/kubelet/pods/3f89ed71-a623-40af-a5b1-5f6333235c14/volumes" Oct 31 03:56:35.604967 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:35.604913 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" path="/var/lib/kubelet/pods/d61881b1-0191-43d7-9ac2-c060af4e8f6c/volumes" Oct 31 03:56:35.722120 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:35.722080 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:35.867763 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:35.864427 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:36.090812 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:36.090769 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:36.131835 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:36.131757 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:36.132336 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:36.132315 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:36.133285 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:36.133261 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:36.861004 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:36.860957 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:37.528630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:37.528592 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" containerID="cri-o://f3bd1d126c4501d8caa00f35ce4915a45cce1f16610bd20d4ca66da3d7706806" gracePeriod=45 Oct 31 03:56:37.545042 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:37.544991 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:37.546789 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:37.546751 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" containerID="cri-o://3be94f0b6a094534c49f97c3d94fb16e081ec07f1e0e0c3155615f216f11ee30" gracePeriod=45 Oct 31 03:56:37.691541 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:37.690769 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:37.879734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:37.879092 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:37.901667 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:37.901636 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:56:37.933356 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:37.933326 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:56:38.012212 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.012171 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:38.018197 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.018156 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-home\" (UniqueName: \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-home\") pod \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " Oct 31 03:56:38.018358 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.018229 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-model-cache\") pod \"51e4007d-c749-49c9-b6b2-301e57a795b3\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " Oct 31 03:56:38.018358 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.018280 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-kube-api-access-sk4wl\" (UniqueName: \"kubernetes.io/projected/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-kube-api-access-sk4wl\") pod \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " Oct 31 03:56:38.018358 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.018338 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-tmp-dir\") pod \"51e4007d-c749-49c9-b6b2-301e57a795b3\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " Oct 31 03:56:38.018597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.018559 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e4007d-c749-49c9-b6b2-301e57a795b3-kube-api-access-hcjsl\" (UniqueName: \"kubernetes.io/projected/51e4007d-c749-49c9-b6b2-301e57a795b3-kube-api-access-hcjsl\") pod \"51e4007d-c749-49c9-b6b2-301e57a795b3\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " Oct 31 03:56:38.018657 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.018617 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51e4007d-c749-49c9-b6b2-301e57a795b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/51e4007d-c749-49c9-b6b2-301e57a795b3-tls-certs\") pod \"51e4007d-c749-49c9-b6b2-301e57a795b3\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " Oct 31 03:56:38.019115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.018656 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-dshm\") pod \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " Oct 31 03:56:38.019115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.018735 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-home\" (UniqueName: \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-home\") pod \"51e4007d-c749-49c9-b6b2-301e57a795b3\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " Oct 31 03:56:38.019115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.018773 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-tls-certs\") pod \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " Oct 31 03:56:38.019115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.018825 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-model-cache\") pod \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " Oct 31 03:56:38.019115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.018872 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-dshm\") pod \"51e4007d-c749-49c9-b6b2-301e57a795b3\" (UID: \"51e4007d-c749-49c9-b6b2-301e57a795b3\") " Oct 31 03:56:38.019115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.018904 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-tmp-dir\") pod \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\" (UID: \"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66\") " Oct 31 03:56:38.019796 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.019769 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-tmp-dir" pod "1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" (UID: "1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:38.019983 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.019966 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-home" pod "1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" (UID: "1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:38.020791 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.020755 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-model-cache" pod "1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" (UID: "1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:38.021125 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.021082 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-home" pod "51e4007d-c749-49c9-b6b2-301e57a795b3" (UID: "51e4007d-c749-49c9-b6b2-301e57a795b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:38.021708 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.021661 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-model-cache" pod "51e4007d-c749-49c9-b6b2-301e57a795b3" (UID: "51e4007d-c749-49c9-b6b2-301e57a795b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:38.022048 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.022028 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-tmp-dir" pod "51e4007d-c749-49c9-b6b2-301e57a795b3" (UID: "51e4007d-c749-49c9-b6b2-301e57a795b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:38.024337 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.024313 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-dshm" pod "1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" (UID: "1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:38.024486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.024330 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-kube-api-access-sk4wl" pod "1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" (UID: "1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66"). InnerVolumeSpecName "kube-api-access-sk4wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:38.024486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.024338 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-tls-certs" pod "1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" (UID: "1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:38.025100 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.025065 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51e4007d-c749-49c9-b6b2-301e57a795b3-tls-certs" pod "51e4007d-c749-49c9-b6b2-301e57a795b3" (UID: "51e4007d-c749-49c9-b6b2-301e57a795b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:38.025516 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.025491 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e4007d-c749-49c9-b6b2-301e57a795b3-kube-api-access-hcjsl" pod "51e4007d-c749-49c9-b6b2-301e57a795b3" (UID: "51e4007d-c749-49c9-b6b2-301e57a795b3"). InnerVolumeSpecName "kube-api-access-hcjsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:38.026548 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.025722 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-dshm" pod "51e4007d-c749-49c9-b6b2-301e57a795b3" (UID: "51e4007d-c749-49c9-b6b2-301e57a795b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:38.120883 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.120776 5490 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-home\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.120883 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.120823 5490 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-tls-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.120883 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.120839 5490 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-model-cache\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.120883 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.120853 5490 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-dshm\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.120883 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.120868 5490 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-tmp-dir\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.120883 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.120881 5490 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-home\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.121345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.120893 5490 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-model-cache\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.121345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.120908 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk4wl\" (UniqueName: \"kubernetes.io/projected/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-kube-api-access-sk4wl\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.121345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.120922 5490 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e4007d-c749-49c9-b6b2-301e57a795b3-tmp-dir\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.121345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.120935 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcjsl\" (UniqueName: \"kubernetes.io/projected/51e4007d-c749-49c9-b6b2-301e57a795b3-kube-api-access-hcjsl\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.121345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.120948 5490 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e4007d-c749-49c9-b6b2-301e57a795b3-tls-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.121345 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.120960 5490 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66-dshm\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:38.280928 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.280889 5490 generic.go:356] "Generic (PLEG): container finished" podID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerID="f3bd1d126c4501d8caa00f35ce4915a45cce1f16610bd20d4ca66da3d7706806" exitCode=0 Oct 31 03:56:38.281129 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.280971 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" Oct 31 03:56:38.281129 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.280993 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" event={"ID":"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66","Type":"ContainerDied","Data":"f3bd1d126c4501d8caa00f35ce4915a45cce1f16610bd20d4ca66da3d7706806"} Oct 31 03:56:38.281129 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.281041 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5" event={"ID":"1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66","Type":"ContainerDied","Data":"fa1b029fa12002d7aa4c989d568457cb502f01904800f45dcbb64a41100bea0b"} Oct 31 03:56:38.281129 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.281060 5490 scope.go:122] "RemoveContainer" containerID="f3bd1d126c4501d8caa00f35ce4915a45cce1f16610bd20d4ca66da3d7706806" Oct 31 03:56:38.283925 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.283866 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" event={"ID":"51e4007d-c749-49c9-b6b2-301e57a795b3","Type":"ContainerDied","Data":"3be94f0b6a094534c49f97c3d94fb16e081ec07f1e0e0c3155615f216f11ee30"} Oct 31 03:56:38.284084 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.284069 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" Oct 31 03:56:38.284649 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.283670 5490 generic.go:356] "Generic (PLEG): container finished" podID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerID="3be94f0b6a094534c49f97c3d94fb16e081ec07f1e0e0c3155615f216f11ee30" exitCode=0 Oct 31 03:56:38.285032 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.284983 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll" event={"ID":"51e4007d-c749-49c9-b6b2-301e57a795b3","Type":"ContainerDied","Data":"b7f6023e0738047ef7a93100ae24364e6245dee56d3a402f3fba1032ad45ac91"} Oct 31 03:56:38.304812 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.304785 5490 scope.go:122] "RemoveContainer" containerID="f3bd1d126c4501d8caa00f35ce4915a45cce1f16610bd20d4ca66da3d7706806" Oct 31 03:56:38.305194 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:38.305170 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3bd1d126c4501d8caa00f35ce4915a45cce1f16610bd20d4ca66da3d7706806\": container with ID starting with f3bd1d126c4501d8caa00f35ce4915a45cce1f16610bd20d4ca66da3d7706806 not found: ID does not exist" containerID="f3bd1d126c4501d8caa00f35ce4915a45cce1f16610bd20d4ca66da3d7706806" Oct 31 03:56:38.305290 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.305205 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3bd1d126c4501d8caa00f35ce4915a45cce1f16610bd20d4ca66da3d7706806"} err="failed to get container status \"f3bd1d126c4501d8caa00f35ce4915a45cce1f16610bd20d4ca66da3d7706806\": rpc error: code = NotFound desc = could not find container \"f3bd1d126c4501d8caa00f35ce4915a45cce1f16610bd20d4ca66da3d7706806\": container with ID starting with f3bd1d126c4501d8caa00f35ce4915a45cce1f16610bd20d4ca66da3d7706806 not found: ID does not exist" Oct 31 03:56:38.305290 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.305223 5490 scope.go:122] "RemoveContainer" containerID="3be94f0b6a094534c49f97c3d94fb16e081ec07f1e0e0c3155615f216f11ee30" Oct 31 03:56:38.316145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.316118 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5"] Oct 31 03:56:38.320605 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.320576 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/distinct-2-w4-dda5cb-kserve-55ddc79b89-s8sj5"] Oct 31 03:56:38.323524 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.323398 5490 scope.go:122] "RemoveContainer" containerID="3be94f0b6a094534c49f97c3d94fb16e081ec07f1e0e0c3155615f216f11ee30" Oct 31 03:56:38.323933 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:38.323902 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3be94f0b6a094534c49f97c3d94fb16e081ec07f1e0e0c3155615f216f11ee30\": container with ID starting with 3be94f0b6a094534c49f97c3d94fb16e081ec07f1e0e0c3155615f216f11ee30 not found: ID does not exist" containerID="3be94f0b6a094534c49f97c3d94fb16e081ec07f1e0e0c3155615f216f11ee30" Oct 31 03:56:38.324096 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.323943 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3be94f0b6a094534c49f97c3d94fb16e081ec07f1e0e0c3155615f216f11ee30"} err="failed to get container status \"3be94f0b6a094534c49f97c3d94fb16e081ec07f1e0e0c3155615f216f11ee30\": rpc error: code = NotFound desc = could not find container \"3be94f0b6a094534c49f97c3d94fb16e081ec07f1e0e0c3155615f216f11ee30\": container with ID starting with 3be94f0b6a094534c49f97c3d94fb16e081ec07f1e0e0c3155615f216f11ee30 not found: ID does not exist" Oct 31 03:56:38.334715 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.334666 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll"] Oct 31 03:56:38.340029 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.340001 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-dda5cb/premium-w4-dda5cb-kserve-796ff5cf99-g2qll"] Oct 31 03:56:38.346603 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.346455 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:38.825792 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:38.825600 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:39.578763 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:39.578722 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" path="/var/lib/kubelet/pods/1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66/volumes" Oct 31 03:56:39.579437 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:39.579418 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" path="/var/lib/kubelet/pods/51e4007d-c749-49c9-b6b2-301e57a795b3/volumes" Oct 31 03:56:39.794906 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:39.794865 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:40.587371 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:40.587321 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:41.181758 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:41.181710 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:42.059197 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:42.059158 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:42.519756 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:42.519718 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf"] Oct 31 03:56:42.519998 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:42.519966 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" podUID="c809aea3-9195-4f90-9ffb-ca153af1a5be" containerName="maas-api" containerID="cri-o://6b5b8525fb106f3dcb186e908c507929a427d3baf29f5efb12f872eccc7b0ffd" gracePeriod=30 Oct 31 03:56:42.614487 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:42.614448 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:42.760721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:42.760589 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:42.885341 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:42.885198 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" Oct 31 03:56:42.910069 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:42.910030 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:42.991307 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:42.991269 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c809aea3-9195-4f90-9ffb-ca153af1a5be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c809aea3-9195-4f90-9ffb-ca153af1a5be-openshift-service-ca\") pod \"c809aea3-9195-4f90-9ffb-ca153af1a5be\" (UID: \"c809aea3-9195-4f90-9ffb-ca153af1a5be\") " Oct 31 03:56:42.991526 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:42.991473 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c809aea3-9195-4f90-9ffb-ca153af1a5be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c809aea3-9195-4f90-9ffb-ca153af1a5be-maas-api-tls\") pod \"c809aea3-9195-4f90-9ffb-ca153af1a5be\" (UID: \"c809aea3-9195-4f90-9ffb-ca153af1a5be\") " Oct 31 03:56:42.991807 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:42.991776 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c809aea3-9195-4f90-9ffb-ca153af1a5be-openshift-service-ca" pod "c809aea3-9195-4f90-9ffb-ca153af1a5be" (UID: "c809aea3-9195-4f90-9ffb-ca153af1a5be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:42.991956 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:42.991932 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c809aea3-9195-4f90-9ffb-ca153af1a5be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c809aea3-9195-4f90-9ffb-ca153af1a5be-metrics-certs\") pod \"c809aea3-9195-4f90-9ffb-ca153af1a5be\" (UID: \"c809aea3-9195-4f90-9ffb-ca153af1a5be\") " Oct 31 03:56:42.992471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:42.992117 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c809aea3-9195-4f90-9ffb-ca153af1a5be-kube-api-access-k4nf2\" (UniqueName: \"kubernetes.io/projected/c809aea3-9195-4f90-9ffb-ca153af1a5be-kube-api-access-k4nf2\") pod \"c809aea3-9195-4f90-9ffb-ca153af1a5be\" (UID: \"c809aea3-9195-4f90-9ffb-ca153af1a5be\") " Oct 31 03:56:42.993175 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:42.993148 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c809aea3-9195-4f90-9ffb-ca153af1a5be-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:42.995121 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:42.995093 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c809aea3-9195-4f90-9ffb-ca153af1a5be-maas-api-tls" pod "c809aea3-9195-4f90-9ffb-ca153af1a5be" (UID: "c809aea3-9195-4f90-9ffb-ca153af1a5be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:42.995537 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:42.995512 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c809aea3-9195-4f90-9ffb-ca153af1a5be-metrics-certs" pod "c809aea3-9195-4f90-9ffb-ca153af1a5be" (UID: "c809aea3-9195-4f90-9ffb-ca153af1a5be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:42.995625 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:42.995571 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c809aea3-9195-4f90-9ffb-ca153af1a5be-kube-api-access-k4nf2" pod "c809aea3-9195-4f90-9ffb-ca153af1a5be" (UID: "c809aea3-9195-4f90-9ffb-ca153af1a5be"). InnerVolumeSpecName "kube-api-access-k4nf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:43.072296 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:43.072255 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:43.109126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:43.104906 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c809aea3-9195-4f90-9ffb-ca153af1a5be-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:43.109126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:43.104948 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c809aea3-9195-4f90-9ffb-ca153af1a5be-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:43.109126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:43.104967 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4nf2\" (UniqueName: \"kubernetes.io/projected/c809aea3-9195-4f90-9ffb-ca153af1a5be-kube-api-access-k4nf2\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:43.335251 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:43.335201 5490 generic.go:356] "Generic (PLEG): container finished" podID="c809aea3-9195-4f90-9ffb-ca153af1a5be" containerID="6b5b8525fb106f3dcb186e908c507929a427d3baf29f5efb12f872eccc7b0ffd" exitCode=0 Oct 31 03:56:43.335251 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:43.335250 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" event={"ID":"c809aea3-9195-4f90-9ffb-ca153af1a5be","Type":"ContainerDied","Data":"6b5b8525fb106f3dcb186e908c507929a427d3baf29f5efb12f872eccc7b0ffd"} Oct 31 03:56:43.335516 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:43.335267 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" Oct 31 03:56:43.335516 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:43.335297 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf" event={"ID":"c809aea3-9195-4f90-9ffb-ca153af1a5be","Type":"ContainerDied","Data":"4435cc1da3eb892efdb59cbc95418ca8432d59c7de9ae87c7fb1934a1d5a0062"} Oct 31 03:56:43.335516 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:43.335325 5490 scope.go:122] "RemoveContainer" containerID="6b5b8525fb106f3dcb186e908c507929a427d3baf29f5efb12f872eccc7b0ffd" Oct 31 03:56:43.348316 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:43.348292 5490 scope.go:122] "RemoveContainer" containerID="6b5b8525fb106f3dcb186e908c507929a427d3baf29f5efb12f872eccc7b0ffd" Oct 31 03:56:43.348647 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:43.348620 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b5b8525fb106f3dcb186e908c507929a427d3baf29f5efb12f872eccc7b0ffd\": container with ID starting with 6b5b8525fb106f3dcb186e908c507929a427d3baf29f5efb12f872eccc7b0ffd not found: ID does not exist" containerID="6b5b8525fb106f3dcb186e908c507929a427d3baf29f5efb12f872eccc7b0ffd" Oct 31 03:56:43.348760 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:43.348655 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b5b8525fb106f3dcb186e908c507929a427d3baf29f5efb12f872eccc7b0ffd"} err="failed to get container status \"6b5b8525fb106f3dcb186e908c507929a427d3baf29f5efb12f872eccc7b0ffd\": rpc error: code = NotFound desc = could not find container \"6b5b8525fb106f3dcb186e908c507929a427d3baf29f5efb12f872eccc7b0ffd\": container with ID starting with 6b5b8525fb106f3dcb186e908c507929a427d3baf29f5efb12f872eccc7b0ffd not found: ID does not exist" Oct 31 03:56:43.361687 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:43.361604 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:43.373858 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:43.373826 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf"] Oct 31 03:56:43.382536 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:43.382508 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d04aad-5dbfb66856-hqlsf"] Oct 31 03:56:43.556254 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:43.556211 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:43.578135 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:43.578104 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c809aea3-9195-4f90-9ffb-ca153af1a5be" path="/var/lib/kubelet/pods/c809aea3-9195-4f90-9ffb-ca153af1a5be/volumes" Oct 31 03:56:43.849152 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:43.849112 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6"] Oct 31 03:56:43.849409 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:43.849378 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" podUID="d2a15588-0492-4ccc-96cc-ab09e1e1402d" containerName="maas-api" containerID="cri-o://a197cd4f451c0d78160082e2f7f2926831f3c17d475609a4a452eeeb0ac098a1" gracePeriod=30 Oct 31 03:56:43.940321 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:43.940280 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:44.171705 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.163894 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" Oct 31 03:56:44.178641 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.178599 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:44.227031 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.226985 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2a15588-0492-4ccc-96cc-ab09e1e1402d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a15588-0492-4ccc-96cc-ab09e1e1402d-maas-api-tls\") pod \"d2a15588-0492-4ccc-96cc-ab09e1e1402d\" (UID: \"d2a15588-0492-4ccc-96cc-ab09e1e1402d\") " Oct 31 03:56:44.227198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.227124 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2a15588-0492-4ccc-96cc-ab09e1e1402d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a15588-0492-4ccc-96cc-ab09e1e1402d-metrics-certs\") pod \"d2a15588-0492-4ccc-96cc-ab09e1e1402d\" (UID: \"d2a15588-0492-4ccc-96cc-ab09e1e1402d\") " Oct 31 03:56:44.227240 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.227203 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a15588-0492-4ccc-96cc-ab09e1e1402d-kube-api-access-flqx7\" (UniqueName: \"kubernetes.io/projected/d2a15588-0492-4ccc-96cc-ab09e1e1402d-kube-api-access-flqx7\") pod \"d2a15588-0492-4ccc-96cc-ab09e1e1402d\" (UID: \"d2a15588-0492-4ccc-96cc-ab09e1e1402d\") " Oct 31 03:56:44.227299 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.227241 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2a15588-0492-4ccc-96cc-ab09e1e1402d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a15588-0492-4ccc-96cc-ab09e1e1402d-openshift-service-ca\") pod \"d2a15588-0492-4ccc-96cc-ab09e1e1402d\" (UID: \"d2a15588-0492-4ccc-96cc-ab09e1e1402d\") " Oct 31 03:56:44.228497 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.228466 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a15588-0492-4ccc-96cc-ab09e1e1402d-openshift-service-ca" pod "d2a15588-0492-4ccc-96cc-ab09e1e1402d" (UID: "d2a15588-0492-4ccc-96cc-ab09e1e1402d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:44.232205 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.232175 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a15588-0492-4ccc-96cc-ab09e1e1402d-metrics-certs" pod "d2a15588-0492-4ccc-96cc-ab09e1e1402d" (UID: "d2a15588-0492-4ccc-96cc-ab09e1e1402d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:44.239180 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.239142 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a15588-0492-4ccc-96cc-ab09e1e1402d-kube-api-access-flqx7" pod "d2a15588-0492-4ccc-96cc-ab09e1e1402d" (UID: "d2a15588-0492-4ccc-96cc-ab09e1e1402d"). InnerVolumeSpecName "kube-api-access-flqx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:44.243797 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.243743 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a15588-0492-4ccc-96cc-ab09e1e1402d-maas-api-tls" pod "d2a15588-0492-4ccc-96cc-ab09e1e1402d" (UID: "d2a15588-0492-4ccc-96cc-ab09e1e1402d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:44.328652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.328610 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:44.328871 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.328804 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a15588-0492-4ccc-96cc-ab09e1e1402d-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:44.328871 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.328836 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a15588-0492-4ccc-96cc-ab09e1e1402d-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:44.328871 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.328849 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flqx7\" (UniqueName: \"kubernetes.io/projected/d2a15588-0492-4ccc-96cc-ab09e1e1402d-kube-api-access-flqx7\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:44.328871 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.328860 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a15588-0492-4ccc-96cc-ab09e1e1402d-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:44.347768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.347733 5490 generic.go:356] "Generic (PLEG): container finished" podID="d2a15588-0492-4ccc-96cc-ab09e1e1402d" containerID="a197cd4f451c0d78160082e2f7f2926831f3c17d475609a4a452eeeb0ac098a1" exitCode=0 Oct 31 03:56:44.347952 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.347832 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" event={"ID":"d2a15588-0492-4ccc-96cc-ab09e1e1402d","Type":"ContainerDied","Data":"a197cd4f451c0d78160082e2f7f2926831f3c17d475609a4a452eeeb0ac098a1"} Oct 31 03:56:44.347952 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.347885 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" Oct 31 03:56:44.347952 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.347894 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6" event={"ID":"d2a15588-0492-4ccc-96cc-ab09e1e1402d","Type":"ContainerDied","Data":"54692745dadfaf314648e822ff4f5685b80d7fd0dac5e443afc39fc52fcd66bb"} Oct 31 03:56:44.347952 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.347929 5490 scope.go:122] "RemoveContainer" containerID="a197cd4f451c0d78160082e2f7f2926831f3c17d475609a4a452eeeb0ac098a1" Oct 31 03:56:44.366149 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.366119 5490 scope.go:122] "RemoveContainer" containerID="a197cd4f451c0d78160082e2f7f2926831f3c17d475609a4a452eeeb0ac098a1" Oct 31 03:56:44.366472 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:44.366451 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a197cd4f451c0d78160082e2f7f2926831f3c17d475609a4a452eeeb0ac098a1\": container with ID starting with a197cd4f451c0d78160082e2f7f2926831f3c17d475609a4a452eeeb0ac098a1 not found: ID does not exist" containerID="a197cd4f451c0d78160082e2f7f2926831f3c17d475609a4a452eeeb0ac098a1" Oct 31 03:56:44.366547 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.366483 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a197cd4f451c0d78160082e2f7f2926831f3c17d475609a4a452eeeb0ac098a1"} err="failed to get container status \"a197cd4f451c0d78160082e2f7f2926831f3c17d475609a4a452eeeb0ac098a1\": rpc error: code = NotFound desc = could not find container \"a197cd4f451c0d78160082e2f7f2926831f3c17d475609a4a452eeeb0ac098a1\": container with ID starting with a197cd4f451c0d78160082e2f7f2926831f3c17d475609a4a452eeeb0ac098a1 not found: ID does not exist" Oct 31 03:56:44.389556 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.389524 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6"] Oct 31 03:56:44.395846 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.395818 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-dda5cb-6474bc868b-b2dx6"] Oct 31 03:56:44.494729 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.494042 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:44.874979 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:44.874933 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:45.097783 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.097742 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr"] Oct 31 03:56:45.098106 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.098053 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" podUID="669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0" containerName="maas-api" containerID="cri-o://47b429fcdff7c912f021dc68ed903c6f35dacd6b21c968a65d612fd15946b68f" gracePeriod=30 Oct 31 03:56:45.190324 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.190284 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:45.386493 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.386461 5490 generic.go:356] "Generic (PLEG): container finished" podID="669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0" containerID="47b429fcdff7c912f021dc68ed903c6f35dacd6b21c968a65d612fd15946b68f" exitCode=0 Oct 31 03:56:45.386646 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.386543 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" event={"ID":"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0","Type":"ContainerDied","Data":"47b429fcdff7c912f021dc68ed903c6f35dacd6b21c968a65d612fd15946b68f"} Oct 31 03:56:45.389871 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.389331 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:45.437721 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.431354 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" Oct 31 03:56:45.538743 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.538699 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:45.575790 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.575758 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-openshift-service-ca\") pod \"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0\" (UID: \"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0\") " Oct 31 03:56:45.575975 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.575827 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-metrics-certs\") pod \"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0\" (UID: \"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0\") " Oct 31 03:56:45.575975 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.575890 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-kube-api-access-brcm7\" (UniqueName: \"kubernetes.io/projected/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-kube-api-access-brcm7\") pod \"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0\" (UID: \"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0\") " Oct 31 03:56:45.575975 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.575944 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-maas-api-tls\") pod \"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0\" (UID: \"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0\") " Oct 31 03:56:45.576260 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.576231 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-openshift-service-ca" pod "669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0" (UID: "669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:45.576390 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.576373 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:45.577994 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.577967 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2a15588-0492-4ccc-96cc-ab09e1e1402d" path="/var/lib/kubelet/pods/d2a15588-0492-4ccc-96cc-ab09e1e1402d/volumes" Oct 31 03:56:45.578880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.578853 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-metrics-certs" pod "669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0" (UID: "669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:45.579028 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.579003 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-maas-api-tls" pod "669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0" (UID: "669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:56:45.579071 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.579010 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-kube-api-access-brcm7" pod "669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0" (UID: "669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0"). InnerVolumeSpecName "kube-api-access-brcm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:45.677363 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.677324 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:45.677363 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.677358 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brcm7\" (UniqueName: \"kubernetes.io/projected/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-kube-api-access-brcm7\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:45.677363 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.677369 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:45.819720 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.819663 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:45.967504 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:45.967410 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:46.223173 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:46.223074 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:46.388346 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:46.388304 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:46.394929 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:46.394902 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" Oct 31 03:56:46.395104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:46.394940 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr" event={"ID":"669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0","Type":"ContainerDied","Data":"904e52b38039b2773f2cedc6bf11ba6e2079373fdd88d2172719f1716f30e37f"} Oct 31 03:56:46.395104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:46.395006 5490 scope.go:122] "RemoveContainer" containerID="47b429fcdff7c912f021dc68ed903c6f35dacd6b21c968a65d612fd15946b68f" Oct 31 03:56:46.419347 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:46.419306 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr"] Oct 31 03:56:46.423189 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:46.423164 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d86a77-589bffc5f4-7m5mr"] Oct 31 03:56:46.502750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:46.502413 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v"] Oct 31 03:56:46.502750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:46.502485 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6"] Oct 31 03:56:46.502750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:46.502691 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" containerID="cri-o://366e1f7adbc32af7df7d579b46e5dbc7616a15291c03d27b78295a678b5b5977" gracePeriod=30 Oct 31 03:56:46.503069 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:46.503011 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" podUID="ca122c09-e327-4371-9879-0870b37db17f" containerName="istio-proxy" containerID="cri-o://9fadd77f20e2703cec2929d2902eeba80251d191aebdab611d75537924d9e080" gracePeriod=30 Oct 31 03:56:46.590256 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:46.590205 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:46.752192 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:46.752148 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:46.908014 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:46.907983 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:47.131412 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:47.131346 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:56:47.491396 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:47.491361 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:47.583225 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:47.583183 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0" path="/var/lib/kubelet/pods/669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0/volumes" Oct 31 03:56:47.708266 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:47.708233 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:48.465834 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:48.465800 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:49.127026 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:49.126959 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:49.315441 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:49.315400 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:56:49.315593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:49.315483 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:56:50.076038 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:50.075992 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:50.287353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:50.287316 5490 scope.go:122] "RemoveContainer" containerID="b047fbd7d43f633432b3e6284e0e4cd0557b8e084c0bb6dabeca99dade16a78b" Oct 31 03:56:50.938088 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:50.938045 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:51.042431 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.042390 5490 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:56:51.042635 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.042448 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" podUID="ca122c09-e327-4371-9879-0870b37db17f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:56:51.203220 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.203127 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:51.343733 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.343695 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:51.476493 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.476391 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59"] Oct 31 03:56:51.476727 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.476693 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" podUID="a1e68997-6d71-4e7e-a80c-ca5c29aa225e" containerName="istio-proxy" containerID="cri-o://e8d475349f1267087117267a60ec79eb175ce4d740bfa1eb2c99d2639f02c72f" gracePeriod=30 Oct 31 03:56:51.576998 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.576038 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:51.773412 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.773321 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:51.832548 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.832525 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:56:51.834656 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.834635 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:56:51.877086 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877052 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-envoy\") pod \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " Oct 31 03:56:51.877086 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877094 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-istio-data\") pod \"ca122c09-e327-4371-9879-0870b37db17f\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " Oct 31 03:56:51.877337 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877116 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-workload-certs\") pod \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " Oct 31 03:56:51.877337 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877141 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-credential-socket\") pod \"ca122c09-e327-4371-9879-0870b37db17f\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " Oct 31 03:56:51.877337 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877189 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca122c09-e327-4371-9879-0870b37db17f-istio-token\" (UniqueName: \"kubernetes.io/projected/ca122c09-e327-4371-9879-0870b37db17f-istio-token\") pod \"ca122c09-e327-4371-9879-0870b37db17f\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " Oct 31 03:56:51.877337 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877219 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-token\" (UniqueName: \"kubernetes.io/projected/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-token\") pod \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " Oct 31 03:56:51.877337 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877265 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-workload-socket\") pod \"ca122c09-e327-4371-9879-0870b37db17f\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " Oct 31 03:56:51.877547 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877336 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca122c09-e327-4371-9879-0870b37db17f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca122c09-e327-4371-9879-0870b37db17f-istiod-ca-cert\") pod \"ca122c09-e327-4371-9879-0870b37db17f\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " Oct 31 03:56:51.877547 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877374 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca122c09-e327-4371-9879-0870b37db17f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca122c09-e327-4371-9879-0870b37db17f-istio-podinfo\") pod \"ca122c09-e327-4371-9879-0870b37db17f\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " Oct 31 03:56:51.877547 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877457 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-workload-socket\") pod \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " Oct 31 03:56:51.877547 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877477 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-istio-data" pod "ca122c09-e327-4371-9879-0870b37db17f" (UID: "ca122c09-e327-4371-9879-0870b37db17f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:51.877547 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877484 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-credential-socket\") pod \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " Oct 31 03:56:51.877547 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877534 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-kube-api-access-964qm\" (UniqueName: \"kubernetes.io/projected/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-kube-api-access-964qm\") pod \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " Oct 31 03:56:51.877943 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877587 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-istio-envoy\") pod \"ca122c09-e327-4371-9879-0870b37db17f\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " Oct 31 03:56:51.877943 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877618 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca122c09-e327-4371-9879-0870b37db17f-kube-api-access-dslk5\" (UniqueName: \"kubernetes.io/projected/ca122c09-e327-4371-9879-0870b37db17f-kube-api-access-dslk5\") pod \"ca122c09-e327-4371-9879-0870b37db17f\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " Oct 31 03:56:51.877943 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877638 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-workload-certs\") pod \"ca122c09-e327-4371-9879-0870b37db17f\" (UID: \"ca122c09-e327-4371-9879-0870b37db17f\") " Oct 31 03:56:51.877943 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877649 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-workload-socket" pod "ca122c09-e327-4371-9879-0870b37db17f" (UID: "ca122c09-e327-4371-9879-0870b37db17f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:51.877943 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877713 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istiod-ca-cert\") pod \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " Oct 31 03:56:51.877943 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877732 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-credential-socket" pod "ca122c09-e327-4371-9879-0870b37db17f" (UID: "ca122c09-e327-4371-9879-0870b37db17f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:51.877943 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877757 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-data\") pod \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " Oct 31 03:56:51.877943 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877802 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-podinfo\") pod \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\" (UID: \"b0596bd0-518a-45ab-8cc1-8ba75268ef6a\") " Oct 31 03:56:51.877943 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.877845 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca122c09-e327-4371-9879-0870b37db17f-istiod-ca-cert" pod "ca122c09-e327-4371-9879-0870b37db17f" (UID: "ca122c09-e327-4371-9879-0870b37db17f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:51.878387 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.878362 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:51.878470 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.878393 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca122c09-e327-4371-9879-0870b37db17f-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:51.878470 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.878408 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:51.878470 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.878424 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:51.879647 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.879622 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca122c09-e327-4371-9879-0870b37db17f-istio-token" pod "ca122c09-e327-4371-9879-0870b37db17f" (UID: "ca122c09-e327-4371-9879-0870b37db17f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:51.880567 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.880048 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-workload-certs" pod "b0596bd0-518a-45ab-8cc1-8ba75268ef6a" (UID: "b0596bd0-518a-45ab-8cc1-8ba75268ef6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:51.880567 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.880277 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-credential-socket" pod "b0596bd0-518a-45ab-8cc1-8ba75268ef6a" (UID: "b0596bd0-518a-45ab-8cc1-8ba75268ef6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:51.880567 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.880290 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-workload-certs" pod "ca122c09-e327-4371-9879-0870b37db17f" (UID: "ca122c09-e327-4371-9879-0870b37db17f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:51.881557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.881530 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istiod-ca-cert" pod "b0596bd0-518a-45ab-8cc1-8ba75268ef6a" (UID: "b0596bd0-518a-45ab-8cc1-8ba75268ef6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:51.882064 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.882038 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-workload-socket" pod "b0596bd0-518a-45ab-8cc1-8ba75268ef6a" (UID: "b0596bd0-518a-45ab-8cc1-8ba75268ef6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:51.882288 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.882264 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-data" pod "b0596bd0-518a-45ab-8cc1-8ba75268ef6a" (UID: "b0596bd0-518a-45ab-8cc1-8ba75268ef6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:51.882366 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.882332 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-istio-envoy" pod "ca122c09-e327-4371-9879-0870b37db17f" (UID: "ca122c09-e327-4371-9879-0870b37db17f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:51.882735 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.882664 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-token" pod "b0596bd0-518a-45ab-8cc1-8ba75268ef6a" (UID: "b0596bd0-518a-45ab-8cc1-8ba75268ef6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:51.882839 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.882806 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-envoy" pod "b0596bd0-518a-45ab-8cc1-8ba75268ef6a" (UID: "b0596bd0-518a-45ab-8cc1-8ba75268ef6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:51.883266 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.883240 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca122c09-e327-4371-9879-0870b37db17f-kube-api-access-dslk5" pod "ca122c09-e327-4371-9879-0870b37db17f" (UID: "ca122c09-e327-4371-9879-0870b37db17f"). InnerVolumeSpecName "kube-api-access-dslk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:51.883416 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.883385 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca122c09-e327-4371-9879-0870b37db17f-istio-podinfo" pod "ca122c09-e327-4371-9879-0870b37db17f" (UID: "ca122c09-e327-4371-9879-0870b37db17f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:51.884151 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.884131 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-podinfo" pod "b0596bd0-518a-45ab-8cc1-8ba75268ef6a" (UID: "b0596bd0-518a-45ab-8cc1-8ba75268ef6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:51.885468 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.885426 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-kube-api-access-964qm" pod "b0596bd0-518a-45ab-8cc1-8ba75268ef6a" (UID: "b0596bd0-518a-45ab-8cc1-8ba75268ef6a"). InnerVolumeSpecName "kube-api-access-964qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:51.979437 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.979394 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca122c09-e327-4371-9879-0870b37db17f-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:51.979437 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.979433 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:51.979437 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.979442 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:51.979815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.979452 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-964qm\" (UniqueName: \"kubernetes.io/projected/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-kube-api-access-964qm\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:51.979815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.979465 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:51.979815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.979474 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dslk5\" (UniqueName: \"kubernetes.io/projected/ca122c09-e327-4371-9879-0870b37db17f-kube-api-access-dslk5\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:51.979815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.979486 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca122c09-e327-4371-9879-0870b37db17f-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:51.979815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.979495 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:51.979815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.979504 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:51.979815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.979515 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:51.979815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.979523 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:51.979815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.979531 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:51.979815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.979539 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca122c09-e327-4371-9879-0870b37db17f-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:51.979815 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:51.979547 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0596bd0-518a-45ab-8cc1-8ba75268ef6a-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:52.123318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.123232 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:52.464964 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.464931 5490 generic.go:356] "Generic (PLEG): container finished" podID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerID="366e1f7adbc32af7df7d579b46e5dbc7616a15291c03d27b78295a678b5b5977" exitCode=0 Oct 31 03:56:52.465471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.465003 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" Oct 31 03:56:52.465471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.465020 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" event={"ID":"b0596bd0-518a-45ab-8cc1-8ba75268ef6a","Type":"ContainerDied","Data":"366e1f7adbc32af7df7d579b46e5dbc7616a15291c03d27b78295a678b5b5977"} Oct 31 03:56:52.465471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.465084 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6" event={"ID":"b0596bd0-518a-45ab-8cc1-8ba75268ef6a","Type":"ContainerDied","Data":"5ec5f08e1e9e6e921d4c7e5b23d557d6f6751386c8ffb7c08b65b941dc069dd7"} Oct 31 03:56:52.465471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.465118 5490 scope.go:122] "RemoveContainer" containerID="366e1f7adbc32af7df7d579b46e5dbc7616a15291c03d27b78295a678b5b5977" Oct 31 03:56:52.466879 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.466853 5490 generic.go:356] "Generic (PLEG): container finished" podID="ca122c09-e327-4371-9879-0870b37db17f" containerID="9fadd77f20e2703cec2929d2902eeba80251d191aebdab611d75537924d9e080" exitCode=0 Oct 31 03:56:52.467024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.466924 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" Oct 31 03:56:52.467024 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.466939 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" event={"ID":"ca122c09-e327-4371-9879-0870b37db17f","Type":"ContainerDied","Data":"9fadd77f20e2703cec2929d2902eeba80251d191aebdab611d75537924d9e080"} Oct 31 03:56:52.467170 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.467026 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v" event={"ID":"ca122c09-e327-4371-9879-0870b37db17f","Type":"ContainerDied","Data":"d8abd9a9d6e9259a3d1ab52982ebafac2b71267ed995f3c76315e17d4fe6e09c"} Oct 31 03:56:52.477602 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.477437 5490 scope.go:122] "RemoveContainer" containerID="f3a2648ff280ec1986d0d43efe9be84cda122733c355195a18480a5c8cd10148" Oct 31 03:56:52.489516 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.489493 5490 scope.go:122] "RemoveContainer" containerID="366e1f7adbc32af7df7d579b46e5dbc7616a15291c03d27b78295a678b5b5977" Oct 31 03:56:52.489870 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:52.489852 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"366e1f7adbc32af7df7d579b46e5dbc7616a15291c03d27b78295a678b5b5977\": container with ID starting with 366e1f7adbc32af7df7d579b46e5dbc7616a15291c03d27b78295a678b5b5977 not found: ID does not exist" containerID="366e1f7adbc32af7df7d579b46e5dbc7616a15291c03d27b78295a678b5b5977" Oct 31 03:56:52.489945 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.489888 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"366e1f7adbc32af7df7d579b46e5dbc7616a15291c03d27b78295a678b5b5977"} err="failed to get container status \"366e1f7adbc32af7df7d579b46e5dbc7616a15291c03d27b78295a678b5b5977\": rpc error: code = NotFound desc = could not find container \"366e1f7adbc32af7df7d579b46e5dbc7616a15291c03d27b78295a678b5b5977\": container with ID starting with 366e1f7adbc32af7df7d579b46e5dbc7616a15291c03d27b78295a678b5b5977 not found: ID does not exist" Oct 31 03:56:52.489945 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.489906 5490 scope.go:122] "RemoveContainer" containerID="f3a2648ff280ec1986d0d43efe9be84cda122733c355195a18480a5c8cd10148" Oct 31 03:56:52.490227 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:52.490211 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a2648ff280ec1986d0d43efe9be84cda122733c355195a18480a5c8cd10148\": container with ID starting with f3a2648ff280ec1986d0d43efe9be84cda122733c355195a18480a5c8cd10148 not found: ID does not exist" containerID="f3a2648ff280ec1986d0d43efe9be84cda122733c355195a18480a5c8cd10148" Oct 31 03:56:52.490294 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.490232 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a2648ff280ec1986d0d43efe9be84cda122733c355195a18480a5c8cd10148"} err="failed to get container status \"f3a2648ff280ec1986d0d43efe9be84cda122733c355195a18480a5c8cd10148\": rpc error: code = NotFound desc = could not find container \"f3a2648ff280ec1986d0d43efe9be84cda122733c355195a18480a5c8cd10148\": container with ID starting with f3a2648ff280ec1986d0d43efe9be84cda122733c355195a18480a5c8cd10148 not found: ID does not exist" Oct 31 03:56:52.490294 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.490248 5490 scope.go:122] "RemoveContainer" containerID="9fadd77f20e2703cec2929d2902eeba80251d191aebdab611d75537924d9e080" Oct 31 03:56:52.500816 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.500790 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6"] Oct 31 03:56:52.503904 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.503880 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-dlgg6"] Oct 31 03:56:52.507379 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.507359 5490 scope.go:122] "RemoveContainer" containerID="9fadd77f20e2703cec2929d2902eeba80251d191aebdab611d75537924d9e080" Oct 31 03:56:52.507725 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:52.507702 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fadd77f20e2703cec2929d2902eeba80251d191aebdab611d75537924d9e080\": container with ID starting with 9fadd77f20e2703cec2929d2902eeba80251d191aebdab611d75537924d9e080 not found: ID does not exist" containerID="9fadd77f20e2703cec2929d2902eeba80251d191aebdab611d75537924d9e080" Oct 31 03:56:52.507786 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.507734 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fadd77f20e2703cec2929d2902eeba80251d191aebdab611d75537924d9e080"} err="failed to get container status \"9fadd77f20e2703cec2929d2902eeba80251d191aebdab611d75537924d9e080\": rpc error: code = NotFound desc = could not find container \"9fadd77f20e2703cec2929d2902eeba80251d191aebdab611d75537924d9e080\": container with ID starting with 9fadd77f20e2703cec2929d2902eeba80251d191aebdab611d75537924d9e080 not found: ID does not exist" Oct 31 03:56:52.512910 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.512887 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v"] Oct 31 03:56:52.517663 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.517636 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-ghv5v"] Oct 31 03:56:52.980828 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:52.980793 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:53.246781 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:53.246662 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:53.405527 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:53.405487 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:53.554284 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:53.554190 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:53.578378 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:53.578344 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" path="/var/lib/kubelet/pods/b0596bd0-518a-45ab-8cc1-8ba75268ef6a/volumes" Oct 31 03:56:53.578926 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:53.578909 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca122c09-e327-4371-9879-0870b37db17f" path="/var/lib/kubelet/pods/ca122c09-e327-4371-9879-0870b37db17f/volumes" Oct 31 03:56:53.715750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:53.712442 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:53.994191 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:53.994143 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:54.192656 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:54.192591 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:54.718991 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:54.718944 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:54.872632 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:54.872561 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:55.008397 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:55.008292 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:55.206270 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:55.206221 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:55.533453 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:55.533421 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:55.738178 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:55.738139 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:55.887817 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:55.883615 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:56.018598 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.018556 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:56.158227 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.158193 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:56.444830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.444738 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:56.754128 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.754035 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:56.813186 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.813094 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:56.885221 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.885187 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:56.931761 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.931720 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw"] Oct 31 03:56:56.938223 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.938189 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr"] Oct 31 03:56:56.950467 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.950431 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5"] Oct 31 03:56:56.964364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.964335 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-data\") pod \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " Oct 31 03:56:56.964561 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.964381 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-podinfo\") pod \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " Oct 31 03:56:56.964561 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.964445 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-envoy\") pod \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " Oct 31 03:56:56.964561 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.964551 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istiod-ca-cert\") pod \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " Oct 31 03:56:56.964774 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.964611 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-kube-api-access-p4k2g\" (UniqueName: \"kubernetes.io/projected/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-kube-api-access-p4k2g\") pod \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " Oct 31 03:56:56.964774 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.964651 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-credential-socket\") pod \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " Oct 31 03:56:56.964774 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.964668 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-data" pod "a1e68997-6d71-4e7e-a80c-ca5c29aa225e" (UID: "a1e68997-6d71-4e7e-a80c-ca5c29aa225e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:56.965126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.965101 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-workload-certs\") pod \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " Oct 31 03:56:56.965339 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.965142 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-workload-socket\") pod \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " Oct 31 03:56:56.965339 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.965170 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-credential-socket" pod "a1e68997-6d71-4e7e-a80c-ca5c29aa225e" (UID: "a1e68997-6d71-4e7e-a80c-ca5c29aa225e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:56.966205 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.965971 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-workload-socket" pod "a1e68997-6d71-4e7e-a80c-ca5c29aa225e" (UID: "a1e68997-6d71-4e7e-a80c-ca5c29aa225e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:56.966205 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.966032 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-workload-certs" pod "a1e68997-6d71-4e7e-a80c-ca5c29aa225e" (UID: "a1e68997-6d71-4e7e-a80c-ca5c29aa225e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:56.966375 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.966161 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istiod-ca-cert" pod "a1e68997-6d71-4e7e-a80c-ca5c29aa225e" (UID: "a1e68997-6d71-4e7e-a80c-ca5c29aa225e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:56:56.966375 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.965220 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-token\" (UniqueName: \"kubernetes.io/projected/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-token\") pod \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\" (UID: \"a1e68997-6d71-4e7e-a80c-ca5c29aa225e\") " Oct 31 03:56:56.966810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.966751 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:56.966810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.966776 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:56.966810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.966791 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:56.966810 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.966804 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:56.967091 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.966820 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:56.968368 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.968336 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-envoy" pod "a1e68997-6d71-4e7e-a80c-ca5c29aa225e" (UID: "a1e68997-6d71-4e7e-a80c-ca5c29aa225e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:56:56.968457 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.968380 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-kube-api-access-p4k2g" pod "a1e68997-6d71-4e7e-a80c-ca5c29aa225e" (UID: "a1e68997-6d71-4e7e-a80c-ca5c29aa225e"). InnerVolumeSpecName "kube-api-access-p4k2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:56.968524 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.968467 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-podinfo" pod "a1e68997-6d71-4e7e-a80c-ca5c29aa225e" (UID: "a1e68997-6d71-4e7e-a80c-ca5c29aa225e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:56:56.969143 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:56.969120 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-token" pod "a1e68997-6d71-4e7e-a80c-ca5c29aa225e" (UID: "a1e68997-6d71-4e7e-a80c-ca5c29aa225e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:56:57.034013 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:57.033907 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:57.068378 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:57.068336 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4k2g\" (UniqueName: \"kubernetes.io/projected/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-kube-api-access-p4k2g\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:57.068378 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:57.068377 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:57.068378 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:57.068387 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:57.068640 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:57.068397 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1e68997-6d71-4e7e-a80c-ca5c29aa225e-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:56:57.167029 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:57.166988 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:57.521652 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:57.521613 5490 generic.go:356] "Generic (PLEG): container finished" podID="a1e68997-6d71-4e7e-a80c-ca5c29aa225e" containerID="e8d475349f1267087117267a60ec79eb175ce4d740bfa1eb2c99d2639f02c72f" exitCode=0 Oct 31 03:56:57.521878 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:57.521750 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" Oct 31 03:56:57.521878 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:57.521748 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" event={"ID":"a1e68997-6d71-4e7e-a80c-ca5c29aa225e","Type":"ContainerDied","Data":"e8d475349f1267087117267a60ec79eb175ce4d740bfa1eb2c99d2639f02c72f"} Oct 31 03:56:57.521878 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:57.521821 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59" event={"ID":"a1e68997-6d71-4e7e-a80c-ca5c29aa225e","Type":"ContainerDied","Data":"f9b221f94fed6c50e9d9c0e7c008058542e90e7dc00169ba5f99e165cc26c55b"} Oct 31 03:56:57.521878 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:57.521857 5490 scope.go:122] "RemoveContainer" containerID="e8d475349f1267087117267a60ec79eb175ce4d740bfa1eb2c99d2639f02c72f" Oct 31 03:56:57.533535 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:57.533511 5490 scope.go:122] "RemoveContainer" containerID="e8d475349f1267087117267a60ec79eb175ce4d740bfa1eb2c99d2639f02c72f" Oct 31 03:56:57.534033 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:56:57.534007 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d475349f1267087117267a60ec79eb175ce4d740bfa1eb2c99d2639f02c72f\": container with ID starting with e8d475349f1267087117267a60ec79eb175ce4d740bfa1eb2c99d2639f02c72f not found: ID does not exist" containerID="e8d475349f1267087117267a60ec79eb175ce4d740bfa1eb2c99d2639f02c72f" Oct 31 03:56:57.534138 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:57.534045 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d475349f1267087117267a60ec79eb175ce4d740bfa1eb2c99d2639f02c72f"} err="failed to get container status \"e8d475349f1267087117267a60ec79eb175ce4d740bfa1eb2c99d2639f02c72f\": rpc error: code = NotFound desc = could not find container \"e8d475349f1267087117267a60ec79eb175ce4d740bfa1eb2c99d2639f02c72f\": container with ID starting with e8d475349f1267087117267a60ec79eb175ce4d740bfa1eb2c99d2639f02c72f not found: ID does not exist" Oct 31 03:56:57.547802 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:57.547771 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59"] Oct 31 03:56:57.552210 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:57.552185 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-sxm59"] Oct 31 03:56:57.585052 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:57.585013 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1e68997-6d71-4e7e-a80c-ca5c29aa225e" path="/var/lib/kubelet/pods/a1e68997-6d71-4e7e-a80c-ca5c29aa225e/volumes" Oct 31 03:56:57.662086 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:57.662049 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:58.278578 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:58.278535 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:58.620472 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:58.620380 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:59.477711 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:59.477637 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:59.664749 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:59.664668 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:56:59.814343 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:56:59.814248 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:00.317257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:00.317222 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:01.106086 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:01.106050 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:01.861700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:01.861154 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:02.322732 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:02.320380 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs"] Oct 31 03:57:02.462205 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:02.462164 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:02.607580 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:02.607484 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:02.745195 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:02.745154 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:03.653912 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:03.653871 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:03.818968 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:03.818927 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:03.973818 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:03.973725 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87"] Oct 31 03:57:03.974257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:03.974225 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" podUID="b0b3f9b0-9f95-4556-87a2-0d7473db75f5" containerName="istio-proxy" containerID="cri-o://f19c8fb69294d695de002163e31d193eb96e5e1d4514f7950d3d3416c22d9302" gracePeriod=30 Oct 31 03:57:04.041575 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:04.041374 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:04.721766 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:04.721723 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:05.515700 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:05.509794 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:05.662010 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:05.661974 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:05.843976 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:05.843869 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:06.212927 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:06.212873 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:06.722416 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:06.722364 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:07.321113 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:07.321000 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" podUID="e9f75494-3ba5-460d-a96d-9ab2f4ffa395" containerName="payload-processing" containerID="cri-o://37213b741452d2ad7a75e8ca947284454ac8bf5a856607630ef1d9adb48c0e8f" gracePeriod=25 Oct 31 03:57:07.661118 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:07.661080 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:08.196617 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:08.196580 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:08.394164 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:08.394123 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:08.559312 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:08.559226 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j"] Oct 31 03:57:08.559534 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:08.559503 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" podUID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerName="istio-proxy" containerID="cri-o://424280cd883a00cc9ef9cb568a8001d97a878f7a2f6c55f4a3e02f39e1fe4787" gracePeriod=30 Oct 31 03:57:08.636981 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:08.636942 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:09.062005 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.061968 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:09.212641 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.211321 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:09.288365 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.288331 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:57:09.440144 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.440102 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-token\" (UniqueName: \"kubernetes.io/projected/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-token\") pod \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " Oct 31 03:57:09.440670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.440188 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-data\") pod \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " Oct 31 03:57:09.440670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.440259 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istiod-ca-cert\") pod \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " Oct 31 03:57:09.440670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.440294 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-workload-socket\") pod \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " Oct 31 03:57:09.440670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.440380 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-credential-socket\") pod \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " Oct 31 03:57:09.440670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.440409 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-envoy\") pod \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " Oct 31 03:57:09.440670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.440437 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-podinfo\") pod \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " Oct 31 03:57:09.440670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.440498 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-workload-certs\") pod \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " Oct 31 03:57:09.440670 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.440527 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-kube-api-access-nr7sj\" (UniqueName: \"kubernetes.io/projected/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-kube-api-access-nr7sj\") pod \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\" (UID: \"b0b3f9b0-9f95-4556-87a2-0d7473db75f5\") " Oct 31 03:57:09.442640 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.440855 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-data" pod "b0b3f9b0-9f95-4556-87a2-0d7473db75f5" (UID: "b0b3f9b0-9f95-4556-87a2-0d7473db75f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:09.442640 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.441063 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.442640 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.441188 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istiod-ca-cert" pod "b0b3f9b0-9f95-4556-87a2-0d7473db75f5" (UID: "b0b3f9b0-9f95-4556-87a2-0d7473db75f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:09.443143 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.443111 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-workload-socket" pod "b0b3f9b0-9f95-4556-87a2-0d7473db75f5" (UID: "b0b3f9b0-9f95-4556-87a2-0d7473db75f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:09.443404 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.443356 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-credential-socket" pod "b0b3f9b0-9f95-4556-87a2-0d7473db75f5" (UID: "b0b3f9b0-9f95-4556-87a2-0d7473db75f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:09.443636 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.443578 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-workload-certs" pod "b0b3f9b0-9f95-4556-87a2-0d7473db75f5" (UID: "b0b3f9b0-9f95-4556-87a2-0d7473db75f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:09.443789 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.443746 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-token" pod "b0b3f9b0-9f95-4556-87a2-0d7473db75f5" (UID: "b0b3f9b0-9f95-4556-87a2-0d7473db75f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:09.443883 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.443857 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-kube-api-access-nr7sj" pod "b0b3f9b0-9f95-4556-87a2-0d7473db75f5" (UID: "b0b3f9b0-9f95-4556-87a2-0d7473db75f5"). InnerVolumeSpecName "kube-api-access-nr7sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:09.444976 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.444947 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-podinfo" pod "b0b3f9b0-9f95-4556-87a2-0d7473db75f5" (UID: "b0b3f9b0-9f95-4556-87a2-0d7473db75f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:57:09.445606 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.445582 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-envoy" pod "b0b3f9b0-9f95-4556-87a2-0d7473db75f5" (UID: "b0b3f9b0-9f95-4556-87a2-0d7473db75f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:09.541856 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.541761 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.541856 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.541797 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr7sj\" (UniqueName: \"kubernetes.io/projected/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-kube-api-access-nr7sj\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.541856 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.541813 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.541856 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.541825 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.541856 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.541838 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.541856 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.541851 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.542763 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.541863 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.542763 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.541878 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b3f9b0-9f95-4556-87a2-0d7473db75f5-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:09.560739 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.560658 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:09.615605 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.615550 5490 generic.go:356] "Generic (PLEG): container finished" podID="b0b3f9b0-9f95-4556-87a2-0d7473db75f5" containerID="f19c8fb69294d695de002163e31d193eb96e5e1d4514f7950d3d3416c22d9302" exitCode=0 Oct 31 03:57:09.615605 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.615601 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" Oct 31 03:57:09.615858 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.615828 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" event={"ID":"b0b3f9b0-9f95-4556-87a2-0d7473db75f5","Type":"ContainerDied","Data":"f19c8fb69294d695de002163e31d193eb96e5e1d4514f7950d3d3416c22d9302"} Oct 31 03:57:09.615908 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.615880 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87" event={"ID":"b0b3f9b0-9f95-4556-87a2-0d7473db75f5","Type":"ContainerDied","Data":"77caa69f9ea2072aeef64449e8ac6e300b1b18e25e5d41c2de15a8a717f48950"} Oct 31 03:57:09.615950 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.615913 5490 scope.go:122] "RemoveContainer" containerID="f19c8fb69294d695de002163e31d193eb96e5e1d4514f7950d3d3416c22d9302" Oct 31 03:57:09.628025 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.627992 5490 scope.go:122] "RemoveContainer" containerID="f19c8fb69294d695de002163e31d193eb96e5e1d4514f7950d3d3416c22d9302" Oct 31 03:57:09.628396 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:57:09.628368 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19c8fb69294d695de002163e31d193eb96e5e1d4514f7950d3d3416c22d9302\": container with ID starting with f19c8fb69294d695de002163e31d193eb96e5e1d4514f7950d3d3416c22d9302 not found: ID does not exist" containerID="f19c8fb69294d695de002163e31d193eb96e5e1d4514f7950d3d3416c22d9302" Oct 31 03:57:09.628481 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.628410 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19c8fb69294d695de002163e31d193eb96e5e1d4514f7950d3d3416c22d9302"} err="failed to get container status \"f19c8fb69294d695de002163e31d193eb96e5e1d4514f7950d3d3416c22d9302\": rpc error: code = NotFound desc = could not find container \"f19c8fb69294d695de002163e31d193eb96e5e1d4514f7950d3d3416c22d9302\": container with ID starting with f19c8fb69294d695de002163e31d193eb96e5e1d4514f7950d3d3416c22d9302 not found: ID does not exist" Oct 31 03:57:09.635562 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.635529 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87"] Oct 31 03:57:09.646668 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:09.646620 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-s7p87"] Oct 31 03:57:10.007938 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:10.007896 5490 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:57:10.008127 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:10.007959 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" podUID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:57:10.192267 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:10.192211 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:10.805426 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:10.805389 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:10.936957 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:10.936919 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:11.578714 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:11.578655 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b3f9b0-9f95-4556-87a2-0d7473db75f5" path="/var/lib/kubelet/pods/b0b3f9b0-9f95-4556-87a2-0d7473db75f5/volumes" Oct 31 03:57:11.736240 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:11.736160 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:11.956121 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:11.956070 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" containerID="cri-o://6ea2af8e4e6ca860f8372354edb18735327dc836ff1970eed9959a69085ac28c" gracePeriod=45 Oct 31 03:57:11.961974 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:11.961931 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" containerID="cri-o://8e0af768643439fb9460606bab9bbb76de89707ee5f82da82ccdc4a365b2c9c4" gracePeriod=45 Oct 31 03:57:11.973773 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:11.973727 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" containerID="cri-o://a2a288cfe831df10f02348c0b21102a6ec902dc025c0219d67fa72f5c78d9279" gracePeriod=45 Oct 31 03:57:12.279430 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.279400 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:57:12.303791 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.303759 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:57:12.310325 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.310293 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:57:12.372186 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.372140 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-home\" (UniqueName: \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-home\") pod \"1c3288c4-4250-4046-921b-4382c4d3ca93\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " Oct 31 03:57:12.373007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.372429 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-home" pod "1c3288c4-4250-4046-921b-4382c4d3ca93" (UID: "1c3288c4-4250-4046-921b-4382c4d3ca93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:12.373007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.372686 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-model-cache" pod "1c3288c4-4250-4046-921b-4382c4d3ca93" (UID: "1c3288c4-4250-4046-921b-4382c4d3ca93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:12.373007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.372280 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-model-cache\") pod \"1c3288c4-4250-4046-921b-4382c4d3ca93\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " Oct 31 03:57:12.373007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.372753 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-model-cache\") pod \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " Oct 31 03:57:12.373007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.372787 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c2b41fc-ff94-4a75-affd-81795799528d-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c2b41fc-ff94-4a75-affd-81795799528d-tls-certs\") pod \"6c2b41fc-ff94-4a75-affd-81795799528d\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " Oct 31 03:57:12.373007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.372827 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-tmp-dir\") pod \"1c3288c4-4250-4046-921b-4382c4d3ca93\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " Oct 31 03:57:12.373007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.372854 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-tmp-dir\") pod \"6c2b41fc-ff94-4a75-affd-81795799528d\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " Oct 31 03:57:12.373007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.372921 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-tls-certs\") pod \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " Oct 31 03:57:12.373007 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.372955 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c3288c4-4250-4046-921b-4382c4d3ca93-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c3288c4-4250-4046-921b-4382c4d3ca93-tls-certs\") pod \"1c3288c4-4250-4046-921b-4382c4d3ca93\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " Oct 31 03:57:12.373611 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.373021 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-home\" (UniqueName: \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-home\") pod \"6c2b41fc-ff94-4a75-affd-81795799528d\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " Oct 31 03:57:12.373611 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.373027 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-model-cache" pod "2b771958-a3d1-40a9-aa74-81b7b33aa1e5" (UID: "2b771958-a3d1-40a9-aa74-81b7b33aa1e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:12.373611 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.373052 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-dshm\") pod \"1c3288c4-4250-4046-921b-4382c4d3ca93\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " Oct 31 03:57:12.373611 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.373121 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-tmp-dir" pod "1c3288c4-4250-4046-921b-4382c4d3ca93" (UID: "1c3288c4-4250-4046-921b-4382c4d3ca93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:12.373611 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.373403 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-home" pod "6c2b41fc-ff94-4a75-affd-81795799528d" (UID: "6c2b41fc-ff94-4a75-affd-81795799528d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:12.373611 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.373537 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-dshm\") pod \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " Oct 31 03:57:12.373611 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.373581 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c3288c4-4250-4046-921b-4382c4d3ca93-kube-api-access-9k2rn\" (UniqueName: \"kubernetes.io/projected/1c3288c4-4250-4046-921b-4382c4d3ca93-kube-api-access-9k2rn\") pod \"1c3288c4-4250-4046-921b-4382c4d3ca93\" (UID: \"1c3288c4-4250-4046-921b-4382c4d3ca93\") " Oct 31 03:57:12.373611 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.373605 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-tmp-dir" pod "6c2b41fc-ff94-4a75-affd-81795799528d" (UID: "6c2b41fc-ff94-4a75-affd-81795799528d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:12.374077 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.373620 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c2b41fc-ff94-4a75-affd-81795799528d-kube-api-access-mqqbt\" (UniqueName: \"kubernetes.io/projected/6c2b41fc-ff94-4a75-affd-81795799528d-kube-api-access-mqqbt\") pod \"6c2b41fc-ff94-4a75-affd-81795799528d\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " Oct 31 03:57:12.374077 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.373652 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-model-cache\") pod \"6c2b41fc-ff94-4a75-affd-81795799528d\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " Oct 31 03:57:12.374077 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.373710 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-dshm\") pod \"6c2b41fc-ff94-4a75-affd-81795799528d\" (UID: \"6c2b41fc-ff94-4a75-affd-81795799528d\") " Oct 31 03:57:12.374077 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.373809 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-kube-api-access-vq45f\" (UniqueName: \"kubernetes.io/projected/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-kube-api-access-vq45f\") pod \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " Oct 31 03:57:12.374077 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.373857 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-tmp-dir\") pod \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " Oct 31 03:57:12.374077 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.373904 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-home\" (UniqueName: \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-home\") pod \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\" (UID: \"2b771958-a3d1-40a9-aa74-81b7b33aa1e5\") " Oct 31 03:57:12.374452 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.374431 5490 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-tmp-dir\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:12.374527 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.374460 5490 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-tmp-dir\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:12.374527 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.374474 5490 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-home\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:12.374527 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.374487 5490 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-home\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:12.374527 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.374502 5490 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-model-cache\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:12.374527 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.374517 5490 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-model-cache\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:12.375249 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.375224 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-model-cache" pod "6c2b41fc-ff94-4a75-affd-81795799528d" (UID: "6c2b41fc-ff94-4a75-affd-81795799528d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:12.376236 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.375589 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-home" pod "2b771958-a3d1-40a9-aa74-81b7b33aa1e5" (UID: "2b771958-a3d1-40a9-aa74-81b7b33aa1e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:12.377274 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.377238 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-tmp-dir" pod "2b771958-a3d1-40a9-aa74-81b7b33aa1e5" (UID: "2b771958-a3d1-40a9-aa74-81b7b33aa1e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:12.378662 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.378634 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-tls-certs" pod "2b771958-a3d1-40a9-aa74-81b7b33aa1e5" (UID: "2b771958-a3d1-40a9-aa74-81b7b33aa1e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:12.378866 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.378843 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-dshm" pod "6c2b41fc-ff94-4a75-affd-81795799528d" (UID: "6c2b41fc-ff94-4a75-affd-81795799528d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:12.380438 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.380341 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-dshm" pod "2b771958-a3d1-40a9-aa74-81b7b33aa1e5" (UID: "2b771958-a3d1-40a9-aa74-81b7b33aa1e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:12.380438 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.380379 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c2b41fc-ff94-4a75-affd-81795799528d-tls-certs" pod "6c2b41fc-ff94-4a75-affd-81795799528d" (UID: "6c2b41fc-ff94-4a75-affd-81795799528d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:12.380438 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.380396 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2b41fc-ff94-4a75-affd-81795799528d-kube-api-access-mqqbt" pod "6c2b41fc-ff94-4a75-affd-81795799528d" (UID: "6c2b41fc-ff94-4a75-affd-81795799528d"). InnerVolumeSpecName "kube-api-access-mqqbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:12.380438 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.380413 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-dshm" pod "1c3288c4-4250-4046-921b-4382c4d3ca93" (UID: "1c3288c4-4250-4046-921b-4382c4d3ca93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:12.380765 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.380445 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c3288c4-4250-4046-921b-4382c4d3ca93-tls-certs" pod "1c3288c4-4250-4046-921b-4382c4d3ca93" (UID: "1c3288c4-4250-4046-921b-4382c4d3ca93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:12.380973 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.380953 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3288c4-4250-4046-921b-4382c4d3ca93-kube-api-access-9k2rn" pod "1c3288c4-4250-4046-921b-4382c4d3ca93" (UID: "1c3288c4-4250-4046-921b-4382c4d3ca93"). InnerVolumeSpecName "kube-api-access-9k2rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:12.381664 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.381643 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-kube-api-access-vq45f" pod "2b771958-a3d1-40a9-aa74-81b7b33aa1e5" (UID: "2b771958-a3d1-40a9-aa74-81b7b33aa1e5"). InnerVolumeSpecName "kube-api-access-vq45f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:12.475533 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.475498 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq45f\" (UniqueName: \"kubernetes.io/projected/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-kube-api-access-vq45f\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:12.475533 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.475530 5490 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-tmp-dir\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:12.475533 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.475541 5490 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-home\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:12.475814 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.475549 5490 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c2b41fc-ff94-4a75-affd-81795799528d-tls-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:12.475814 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.475558 5490 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-tls-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:12.475814 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.475566 5490 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c3288c4-4250-4046-921b-4382c4d3ca93-tls-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:12.475814 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.475573 5490 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c3288c4-4250-4046-921b-4382c4d3ca93-dshm\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:12.475814 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.475581 5490 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b771958-a3d1-40a9-aa74-81b7b33aa1e5-dshm\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:12.475814 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.475589 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k2rn\" (UniqueName: \"kubernetes.io/projected/1c3288c4-4250-4046-921b-4382c4d3ca93-kube-api-access-9k2rn\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:12.475814 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.475600 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqqbt\" (UniqueName: \"kubernetes.io/projected/6c2b41fc-ff94-4a75-affd-81795799528d-kube-api-access-mqqbt\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:12.475814 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.475610 5490 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-model-cache\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:12.475814 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.475618 5490 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c2b41fc-ff94-4a75-affd-81795799528d-dshm\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:12.526962 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.526817 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:12.637505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.637470 5490 generic.go:356] "Generic (PLEG): container finished" podID="6c2b41fc-ff94-4a75-affd-81795799528d" containerID="8e0af768643439fb9460606bab9bbb76de89707ee5f82da82ccdc4a365b2c9c4" exitCode=0 Oct 31 03:57:12.637716 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.637526 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" Oct 31 03:57:12.637716 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.637553 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" event={"ID":"6c2b41fc-ff94-4a75-affd-81795799528d","Type":"ContainerDied","Data":"8e0af768643439fb9460606bab9bbb76de89707ee5f82da82ccdc4a365b2c9c4"} Oct 31 03:57:12.637716 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.637600 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr" event={"ID":"6c2b41fc-ff94-4a75-affd-81795799528d","Type":"ContainerDied","Data":"3a35650bb6112ba2ffa1c03456c4ed9888e54fb6aed498199bc317efc012d242"} Oct 31 03:57:12.637716 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.637613 5490 scope.go:122] "RemoveContainer" containerID="8e0af768643439fb9460606bab9bbb76de89707ee5f82da82ccdc4a365b2c9c4" Oct 31 03:57:12.639166 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.639142 5490 generic.go:356] "Generic (PLEG): container finished" podID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerID="6ea2af8e4e6ca860f8372354edb18735327dc836ff1970eed9959a69085ac28c" exitCode=0 Oct 31 03:57:12.639303 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.639192 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" Oct 31 03:57:12.639303 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.639223 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" event={"ID":"2b771958-a3d1-40a9-aa74-81b7b33aa1e5","Type":"ContainerDied","Data":"6ea2af8e4e6ca860f8372354edb18735327dc836ff1970eed9959a69085ac28c"} Oct 31 03:57:12.639303 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.639266 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw" event={"ID":"2b771958-a3d1-40a9-aa74-81b7b33aa1e5","Type":"ContainerDied","Data":"7d36a300e5d4e2f7aeb96870f1ad3a66e8793b96b403219ac2ee2979052963fc"} Oct 31 03:57:12.640963 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.640937 5490 generic.go:356] "Generic (PLEG): container finished" podID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerID="a2a288cfe831df10f02348c0b21102a6ec902dc025c0219d67fa72f5c78d9279" exitCode=0 Oct 31 03:57:12.641076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.640979 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" Oct 31 03:57:12.641076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.641017 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" event={"ID":"1c3288c4-4250-4046-921b-4382c4d3ca93","Type":"ContainerDied","Data":"a2a288cfe831df10f02348c0b21102a6ec902dc025c0219d67fa72f5c78d9279"} Oct 31 03:57:12.641076 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.641055 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5" event={"ID":"1c3288c4-4250-4046-921b-4382c4d3ca93","Type":"ContainerDied","Data":"68123b46158fd9ec0de8e612476c23f96b1b2e0b82abc78e38cb877a53646524"} Oct 31 03:57:12.650485 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.650403 5490 scope.go:122] "RemoveContainer" containerID="8e0af768643439fb9460606bab9bbb76de89707ee5f82da82ccdc4a365b2c9c4" Oct 31 03:57:12.650806 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:57:12.650778 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e0af768643439fb9460606bab9bbb76de89707ee5f82da82ccdc4a365b2c9c4\": container with ID starting with 8e0af768643439fb9460606bab9bbb76de89707ee5f82da82ccdc4a365b2c9c4 not found: ID does not exist" containerID="8e0af768643439fb9460606bab9bbb76de89707ee5f82da82ccdc4a365b2c9c4" Oct 31 03:57:12.650937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.650814 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e0af768643439fb9460606bab9bbb76de89707ee5f82da82ccdc4a365b2c9c4"} err="failed to get container status \"8e0af768643439fb9460606bab9bbb76de89707ee5f82da82ccdc4a365b2c9c4\": rpc error: code = NotFound desc = could not find container \"8e0af768643439fb9460606bab9bbb76de89707ee5f82da82ccdc4a365b2c9c4\": container with ID starting with 8e0af768643439fb9460606bab9bbb76de89707ee5f82da82ccdc4a365b2c9c4 not found: ID does not exist" Oct 31 03:57:12.650937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.650831 5490 scope.go:122] "RemoveContainer" containerID="6ea2af8e4e6ca860f8372354edb18735327dc836ff1970eed9959a69085ac28c" Oct 31 03:57:12.661512 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.661488 5490 scope.go:122] "RemoveContainer" containerID="6ea2af8e4e6ca860f8372354edb18735327dc836ff1970eed9959a69085ac28c" Oct 31 03:57:12.661798 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:57:12.661780 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea2af8e4e6ca860f8372354edb18735327dc836ff1970eed9959a69085ac28c\": container with ID starting with 6ea2af8e4e6ca860f8372354edb18735327dc836ff1970eed9959a69085ac28c not found: ID does not exist" containerID="6ea2af8e4e6ca860f8372354edb18735327dc836ff1970eed9959a69085ac28c" Oct 31 03:57:12.661862 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.661805 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea2af8e4e6ca860f8372354edb18735327dc836ff1970eed9959a69085ac28c"} err="failed to get container status \"6ea2af8e4e6ca860f8372354edb18735327dc836ff1970eed9959a69085ac28c\": rpc error: code = NotFound desc = could not find container \"6ea2af8e4e6ca860f8372354edb18735327dc836ff1970eed9959a69085ac28c\": container with ID starting with 6ea2af8e4e6ca860f8372354edb18735327dc836ff1970eed9959a69085ac28c not found: ID does not exist" Oct 31 03:57:12.661862 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.661823 5490 scope.go:122] "RemoveContainer" containerID="a2a288cfe831df10f02348c0b21102a6ec902dc025c0219d67fa72f5c78d9279" Oct 31 03:57:12.667831 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.667798 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr"] Oct 31 03:57:12.671269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.671244 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/distinct-w3-090ebd-kserve-f4dd967c6-tslkr"] Oct 31 03:57:12.672653 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.672637 5490 scope.go:122] "RemoveContainer" containerID="a2a288cfe831df10f02348c0b21102a6ec902dc025c0219d67fa72f5c78d9279" Oct 31 03:57:12.673036 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:57:12.673015 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2a288cfe831df10f02348c0b21102a6ec902dc025c0219d67fa72f5c78d9279\": container with ID starting with a2a288cfe831df10f02348c0b21102a6ec902dc025c0219d67fa72f5c78d9279 not found: ID does not exist" containerID="a2a288cfe831df10f02348c0b21102a6ec902dc025c0219d67fa72f5c78d9279" Oct 31 03:57:12.673108 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.673046 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2a288cfe831df10f02348c0b21102a6ec902dc025c0219d67fa72f5c78d9279"} err="failed to get container status \"a2a288cfe831df10f02348c0b21102a6ec902dc025c0219d67fa72f5c78d9279\": rpc error: code = NotFound desc = could not find container \"a2a288cfe831df10f02348c0b21102a6ec902dc025c0219d67fa72f5c78d9279\": container with ID starting with a2a288cfe831df10f02348c0b21102a6ec902dc025c0219d67fa72f5c78d9279 not found: ID does not exist" Oct 31 03:57:12.681923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.681883 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw"] Oct 31 03:57:12.688143 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.688111 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/distinct-2-w3-090ebd-kserve-ccb76bb7-8dhbw"] Oct 31 03:57:12.701304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.701273 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5"] Oct 31 03:57:12.708605 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.708576 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/premium-w3-090ebd-kserve-7dc669c77-s9qp5"] Oct 31 03:57:12.993390 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:12.993353 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:13.197099 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.197054 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:13.576331 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.576283 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" path="/var/lib/kubelet/pods/1c3288c4-4250-4046-921b-4382c4d3ca93/volumes" Oct 31 03:57:13.576775 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.576759 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" path="/var/lib/kubelet/pods/2b771958-a3d1-40a9-aa74-81b7b33aa1e5/volumes" Oct 31 03:57:13.577174 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.577160 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" path="/var/lib/kubelet/pods/6c2b41fc-ff94-4a75-affd-81795799528d/volumes" Oct 31 03:57:13.842768 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.842649 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:57:13.891432 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.891385 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-envoy\") pod \"dbec9363-80a5-497b-8dd8-45b9fba41328\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " Oct 31 03:57:13.891610 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.891506 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbec9363-80a5-497b-8dd8-45b9fba41328-kube-api-access-bk6jg\" (UniqueName: \"kubernetes.io/projected/dbec9363-80a5-497b-8dd8-45b9fba41328-kube-api-access-bk6jg\") pod \"dbec9363-80a5-497b-8dd8-45b9fba41328\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " Oct 31 03:57:13.891610 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.891544 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-data\") pod \"dbec9363-80a5-497b-8dd8-45b9fba41328\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " Oct 31 03:57:13.891763 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.891615 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbec9363-80a5-497b-8dd8-45b9fba41328-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbec9363-80a5-497b-8dd8-45b9fba41328-istiod-ca-cert\") pod \"dbec9363-80a5-497b-8dd8-45b9fba41328\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " Oct 31 03:57:13.892666 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.892509 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-data" pod "dbec9363-80a5-497b-8dd8-45b9fba41328" (UID: "dbec9363-80a5-497b-8dd8-45b9fba41328"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:13.892666 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.892565 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:13.893145 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.893117 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbec9363-80a5-497b-8dd8-45b9fba41328-istiod-ca-cert" pod "dbec9363-80a5-497b-8dd8-45b9fba41328" (UID: "dbec9363-80a5-497b-8dd8-45b9fba41328"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:13.893367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.893172 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-token\" (UniqueName: \"kubernetes.io/projected/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-token\") pod \"dbec9363-80a5-497b-8dd8-45b9fba41328\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " Oct 31 03:57:13.893367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.893216 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-workload-certs\") pod \"dbec9363-80a5-497b-8dd8-45b9fba41328\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " Oct 31 03:57:13.893367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.893296 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-podinfo\") pod \"dbec9363-80a5-497b-8dd8-45b9fba41328\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " Oct 31 03:57:13.893367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.893364 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-credential-socket\") pod \"dbec9363-80a5-497b-8dd8-45b9fba41328\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " Oct 31 03:57:13.893661 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.893393 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-workload-socket\") pod \"dbec9363-80a5-497b-8dd8-45b9fba41328\" (UID: \"dbec9363-80a5-497b-8dd8-45b9fba41328\") " Oct 31 03:57:13.895102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.893694 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-credential-socket" pod "dbec9363-80a5-497b-8dd8-45b9fba41328" (UID: "dbec9363-80a5-497b-8dd8-45b9fba41328"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:13.895102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.893920 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-workload-certs" pod "dbec9363-80a5-497b-8dd8-45b9fba41328" (UID: "dbec9363-80a5-497b-8dd8-45b9fba41328"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:13.895102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.893966 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-workload-socket" pod "dbec9363-80a5-497b-8dd8-45b9fba41328" (UID: "dbec9363-80a5-497b-8dd8-45b9fba41328"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:13.895102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.894271 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:13.895102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.894295 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:13.895102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.894308 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:13.895102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.894322 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbec9363-80a5-497b-8dd8-45b9fba41328-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:13.895102 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.894334 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:13.896744 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.896706 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-token" pod "dbec9363-80a5-497b-8dd8-45b9fba41328" (UID: "dbec9363-80a5-497b-8dd8-45b9fba41328"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:13.898130 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.898103 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbec9363-80a5-497b-8dd8-45b9fba41328-kube-api-access-bk6jg" pod "dbec9363-80a5-497b-8dd8-45b9fba41328" (UID: "dbec9363-80a5-497b-8dd8-45b9fba41328"). InnerVolumeSpecName "kube-api-access-bk6jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:13.898326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.898307 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-envoy" pod "dbec9363-80a5-497b-8dd8-45b9fba41328" (UID: "dbec9363-80a5-497b-8dd8-45b9fba41328"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:13.898516 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.898499 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-podinfo" pod "dbec9363-80a5-497b-8dd8-45b9fba41328" (UID: "dbec9363-80a5-497b-8dd8-45b9fba41328"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:57:13.989424 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.989382 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc"] Oct 31 03:57:13.996218 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.996182 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:13.996218 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.996218 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:13.996749 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.996236 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbec9363-80a5-497b-8dd8-45b9fba41328-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:13.996749 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:13.996250 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk6jg\" (UniqueName: \"kubernetes.io/projected/dbec9363-80a5-497b-8dd8-45b9fba41328-kube-api-access-bk6jg\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:14.141459 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:14.141368 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:14.294409 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:14.294368 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:14.663315 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:14.663275 5490 generic.go:356] "Generic (PLEG): container finished" podID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerID="424280cd883a00cc9ef9cb568a8001d97a878f7a2f6c55f4a3e02f39e1fe4787" exitCode=0 Oct 31 03:57:14.663527 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:14.663330 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" Oct 31 03:57:14.663527 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:14.663334 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" event={"ID":"dbec9363-80a5-497b-8dd8-45b9fba41328","Type":"ContainerDied","Data":"424280cd883a00cc9ef9cb568a8001d97a878f7a2f6c55f4a3e02f39e1fe4787"} Oct 31 03:57:14.663527 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:14.663410 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j" event={"ID":"dbec9363-80a5-497b-8dd8-45b9fba41328","Type":"ContainerDied","Data":"e430b96fd5646df1fcc81003a22f1361cd2fc0f385d15a070971190640671f4e"} Oct 31 03:57:14.663527 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:14.663443 5490 scope.go:122] "RemoveContainer" containerID="424280cd883a00cc9ef9cb568a8001d97a878f7a2f6c55f4a3e02f39e1fe4787" Oct 31 03:57:14.675264 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:14.675237 5490 scope.go:122] "RemoveContainer" containerID="0f07b92fbded13cb25bc3a9fed826a9aa0bf26995e6498bd4d386f0a0ef0d7c9" Oct 31 03:57:14.686054 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:14.685904 5490 scope.go:122] "RemoveContainer" containerID="424280cd883a00cc9ef9cb568a8001d97a878f7a2f6c55f4a3e02f39e1fe4787" Oct 31 03:57:14.686233 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:57:14.686212 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"424280cd883a00cc9ef9cb568a8001d97a878f7a2f6c55f4a3e02f39e1fe4787\": container with ID starting with 424280cd883a00cc9ef9cb568a8001d97a878f7a2f6c55f4a3e02f39e1fe4787 not found: ID does not exist" containerID="424280cd883a00cc9ef9cb568a8001d97a878f7a2f6c55f4a3e02f39e1fe4787" Oct 31 03:57:14.686272 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:14.686245 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"424280cd883a00cc9ef9cb568a8001d97a878f7a2f6c55f4a3e02f39e1fe4787"} err="failed to get container status \"424280cd883a00cc9ef9cb568a8001d97a878f7a2f6c55f4a3e02f39e1fe4787\": rpc error: code = NotFound desc = could not find container \"424280cd883a00cc9ef9cb568a8001d97a878f7a2f6c55f4a3e02f39e1fe4787\": container with ID starting with 424280cd883a00cc9ef9cb568a8001d97a878f7a2f6c55f4a3e02f39e1fe4787 not found: ID does not exist" Oct 31 03:57:14.686272 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:14.686263 5490 scope.go:122] "RemoveContainer" containerID="0f07b92fbded13cb25bc3a9fed826a9aa0bf26995e6498bd4d386f0a0ef0d7c9" Oct 31 03:57:14.686520 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:57:14.686502 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f07b92fbded13cb25bc3a9fed826a9aa0bf26995e6498bd4d386f0a0ef0d7c9\": container with ID starting with 0f07b92fbded13cb25bc3a9fed826a9aa0bf26995e6498bd4d386f0a0ef0d7c9 not found: ID does not exist" containerID="0f07b92fbded13cb25bc3a9fed826a9aa0bf26995e6498bd4d386f0a0ef0d7c9" Oct 31 03:57:14.686563 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:14.686526 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f07b92fbded13cb25bc3a9fed826a9aa0bf26995e6498bd4d386f0a0ef0d7c9"} err="failed to get container status \"0f07b92fbded13cb25bc3a9fed826a9aa0bf26995e6498bd4d386f0a0ef0d7c9\": rpc error: code = NotFound desc = could not find container \"0f07b92fbded13cb25bc3a9fed826a9aa0bf26995e6498bd4d386f0a0ef0d7c9\": container with ID starting with 0f07b92fbded13cb25bc3a9fed826a9aa0bf26995e6498bd4d386f0a0ef0d7c9 not found: ID does not exist" Oct 31 03:57:14.709263 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:14.709221 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j"] Oct 31 03:57:14.718552 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:14.718520 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5396c2-openshift-default-6ccfdf4597-rwj9j"] Oct 31 03:57:14.975886 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:14.975784 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:15.575615 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:15.575565 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbec9363-80a5-497b-8dd8-45b9fba41328" path="/var/lib/kubelet/pods/dbec9363-80a5-497b-8dd8-45b9fba41328/volumes" Oct 31 03:57:15.715424 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:15.715389 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:16.504237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:16.504197 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:16.511812 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:16.511774 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" podUID="e9f75494-3ba5-460d-a96d-9ab2f4ffa395" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:57:17.129742 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:57:17.129703 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:57:17.420346 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:17.420311 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:18.097126 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:18.097089 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:18.407435 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:18.407396 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:18.761432 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:18.761338 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:19.327755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:19.327716 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf"] Oct 31 03:57:19.470959 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:19.470922 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:19.697827 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:19.697515 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:20.294622 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:20.294577 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:21.055395 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:21.055351 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:21.818062 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:21.818026 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:22.594862 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:22.594821 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:23.298954 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:23.298910 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:23.849347 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:23.849311 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd"] Oct 31 03:57:23.849867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:23.849519 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" podUID="288a58cd-9286-422b-bff8-15da47bc1a24" containerName="maas-api" containerID="cri-o://57363db53745908cd4ba258785ed632f5eaabe1f44fa4682c42ca106a42f93b4" gracePeriod=30 Oct 31 03:57:23.951918 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:23.951872 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:24.130559 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.130532 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" Oct 31 03:57:24.219009 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.218971 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/288a58cd-9286-422b-bff8-15da47bc1a24-metrics-certs\" (UniqueName: \"kubernetes.io/secret/288a58cd-9286-422b-bff8-15da47bc1a24-metrics-certs\") pod \"288a58cd-9286-422b-bff8-15da47bc1a24\" (UID: \"288a58cd-9286-422b-bff8-15da47bc1a24\") " Oct 31 03:57:24.219237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.219052 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/288a58cd-9286-422b-bff8-15da47bc1a24-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/288a58cd-9286-422b-bff8-15da47bc1a24-openshift-service-ca\") pod \"288a58cd-9286-422b-bff8-15da47bc1a24\" (UID: \"288a58cd-9286-422b-bff8-15da47bc1a24\") " Oct 31 03:57:24.219237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.219169 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/288a58cd-9286-422b-bff8-15da47bc1a24-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/288a58cd-9286-422b-bff8-15da47bc1a24-maas-api-tls\") pod \"288a58cd-9286-422b-bff8-15da47bc1a24\" (UID: \"288a58cd-9286-422b-bff8-15da47bc1a24\") " Oct 31 03:57:24.219237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.219210 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288a58cd-9286-422b-bff8-15da47bc1a24-kube-api-access-9zxkf\" (UniqueName: \"kubernetes.io/projected/288a58cd-9286-422b-bff8-15da47bc1a24-kube-api-access-9zxkf\") pod \"288a58cd-9286-422b-bff8-15da47bc1a24\" (UID: \"288a58cd-9286-422b-bff8-15da47bc1a24\") " Oct 31 03:57:24.219703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.219652 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288a58cd-9286-422b-bff8-15da47bc1a24-openshift-service-ca" pod "288a58cd-9286-422b-bff8-15da47bc1a24" (UID: "288a58cd-9286-422b-bff8-15da47bc1a24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:24.221946 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.221918 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288a58cd-9286-422b-bff8-15da47bc1a24-metrics-certs" pod "288a58cd-9286-422b-bff8-15da47bc1a24" (UID: "288a58cd-9286-422b-bff8-15da47bc1a24"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:24.222104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.222061 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288a58cd-9286-422b-bff8-15da47bc1a24-maas-api-tls" pod "288a58cd-9286-422b-bff8-15da47bc1a24" (UID: "288a58cd-9286-422b-bff8-15da47bc1a24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:24.222104 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.222080 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288a58cd-9286-422b-bff8-15da47bc1a24-kube-api-access-9zxkf" pod "288a58cd-9286-422b-bff8-15da47bc1a24" (UID: "288a58cd-9286-422b-bff8-15da47bc1a24"). InnerVolumeSpecName "kube-api-access-9zxkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:24.226828 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.226777 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:24.321586 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.321547 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/288a58cd-9286-422b-bff8-15da47bc1a24-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:24.321586 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.321588 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zxkf\" (UniqueName: \"kubernetes.io/projected/288a58cd-9286-422b-bff8-15da47bc1a24-kube-api-access-9zxkf\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:24.321855 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.321602 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/288a58cd-9286-422b-bff8-15da47bc1a24-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:24.321855 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.321611 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/288a58cd-9286-422b-bff8-15da47bc1a24-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:24.328779 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.328737 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" podUID="4cf50264-0d76-46ca-b2cb-998ae3bbeb1b" containerName="payload-processing" containerID="cri-o://9a8d03166a4c32dde0efd5baaefb8adde72ac0c51ba156399e116bc5f90a3b3b" gracePeriod=25 Oct 31 03:57:24.421083 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.421033 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:24.735669 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.735572 5490 generic.go:356] "Generic (PLEG): container finished" podID="288a58cd-9286-422b-bff8-15da47bc1a24" containerID="57363db53745908cd4ba258785ed632f5eaabe1f44fa4682c42ca106a42f93b4" exitCode=0 Oct 31 03:57:24.735669 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.735619 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" Oct 31 03:57:24.735669 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.735656 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" event={"ID":"288a58cd-9286-422b-bff8-15da47bc1a24","Type":"ContainerDied","Data":"57363db53745908cd4ba258785ed632f5eaabe1f44fa4682c42ca106a42f93b4"} Oct 31 03:57:24.736019 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.735728 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd" event={"ID":"288a58cd-9286-422b-bff8-15da47bc1a24","Type":"ContainerDied","Data":"beb262ecbd696cc201614255b98292ef089c7fab793319d21c7901613b2dd7c4"} Oct 31 03:57:24.736019 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.735756 5490 scope.go:122] "RemoveContainer" containerID="57363db53745908cd4ba258785ed632f5eaabe1f44fa4682c42ca106a42f93b4" Oct 31 03:57:24.747021 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.747001 5490 scope.go:122] "RemoveContainer" containerID="57363db53745908cd4ba258785ed632f5eaabe1f44fa4682c42ca106a42f93b4" Oct 31 03:57:24.747339 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:57:24.747319 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57363db53745908cd4ba258785ed632f5eaabe1f44fa4682c42ca106a42f93b4\": container with ID starting with 57363db53745908cd4ba258785ed632f5eaabe1f44fa4682c42ca106a42f93b4 not found: ID does not exist" containerID="57363db53745908cd4ba258785ed632f5eaabe1f44fa4682c42ca106a42f93b4" Oct 31 03:57:24.747413 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.747346 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57363db53745908cd4ba258785ed632f5eaabe1f44fa4682c42ca106a42f93b4"} err="failed to get container status \"57363db53745908cd4ba258785ed632f5eaabe1f44fa4682c42ca106a42f93b4\": rpc error: code = NotFound desc = could not find container \"57363db53745908cd4ba258785ed632f5eaabe1f44fa4682c42ca106a42f93b4\": container with ID starting with 57363db53745908cd4ba258785ed632f5eaabe1f44fa4682c42ca106a42f93b4 not found: ID does not exist" Oct 31 03:57:24.761487 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.761456 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd"] Oct 31 03:57:24.769785 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.769758 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-090ebd-7f64cbcc6d-5sdsd"] Oct 31 03:57:24.805971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:24.805937 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:25.105135 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:25.105010 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:25.576732 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:25.576697 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="288a58cd-9286-422b-bff8-15da47bc1a24" path="/var/lib/kubelet/pods/288a58cd-9286-422b-bff8-15da47bc1a24/volumes" Oct 31 03:57:25.802469 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:25.802433 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:26.470995 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:26.470956 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:26.511509 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:26.511472 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" podUID="e9f75494-3ba5-460d-a96d-9ab2f4ffa395" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:57:27.367004 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:27.366965 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:27.601369 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:27.601326 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" Oct 31 03:57:27.760505 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:27.760463 5490 generic.go:356] "Generic (PLEG): container finished" podID="e9f75494-3ba5-460d-a96d-9ab2f4ffa395" containerID="37213b741452d2ad7a75e8ca947284454ac8bf5a856607630ef1d9adb48c0e8f" exitCode=0 Oct 31 03:57:27.760697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:27.760512 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" Oct 31 03:57:27.760697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:27.760514 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" event={"ID":"e9f75494-3ba5-460d-a96d-9ab2f4ffa395","Type":"ContainerDied","Data":"37213b741452d2ad7a75e8ca947284454ac8bf5a856607630ef1d9adb48c0e8f"} Oct 31 03:57:27.760697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:27.760620 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs" event={"ID":"e9f75494-3ba5-460d-a96d-9ab2f4ffa395","Type":"ContainerDied","Data":"103e2faa078c1eec65101a799b0fff026756c504556ec3d323265645c2dfaf18"} Oct 31 03:57:27.760697 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:27.760651 5490 scope.go:122] "RemoveContainer" containerID="37213b741452d2ad7a75e8ca947284454ac8bf5a856607630ef1d9adb48c0e8f" Oct 31 03:57:27.761151 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:27.761127 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9f75494-3ba5-460d-a96d-9ab2f4ffa395-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f75494-3ba5-460d-a96d-9ab2f4ffa395-plugins-config-volume\") pod \"e9f75494-3ba5-460d-a96d-9ab2f4ffa395\" (UID: \"e9f75494-3ba5-460d-a96d-9ab2f4ffa395\") " Oct 31 03:57:27.761237 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:27.761190 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f75494-3ba5-460d-a96d-9ab2f4ffa395-kube-api-access-9qjhp\" (UniqueName: \"kubernetes.io/projected/e9f75494-3ba5-460d-a96d-9ab2f4ffa395-kube-api-access-9qjhp\") pod \"e9f75494-3ba5-460d-a96d-9ab2f4ffa395\" (UID: \"e9f75494-3ba5-460d-a96d-9ab2f4ffa395\") " Oct 31 03:57:27.761560 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:27.761541 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f75494-3ba5-460d-a96d-9ab2f4ffa395-plugins-config-volume" pod "e9f75494-3ba5-460d-a96d-9ab2f4ffa395" (UID: "e9f75494-3ba5-460d-a96d-9ab2f4ffa395"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:27.763525 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:27.763495 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f75494-3ba5-460d-a96d-9ab2f4ffa395-kube-api-access-9qjhp" pod "e9f75494-3ba5-460d-a96d-9ab2f4ffa395" (UID: "e9f75494-3ba5-460d-a96d-9ab2f4ffa395"). InnerVolumeSpecName "kube-api-access-9qjhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:27.778561 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:27.778536 5490 scope.go:122] "RemoveContainer" containerID="37213b741452d2ad7a75e8ca947284454ac8bf5a856607630ef1d9adb48c0e8f" Oct 31 03:57:27.778911 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:57:27.778891 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37213b741452d2ad7a75e8ca947284454ac8bf5a856607630ef1d9adb48c0e8f\": container with ID starting with 37213b741452d2ad7a75e8ca947284454ac8bf5a856607630ef1d9adb48c0e8f not found: ID does not exist" containerID="37213b741452d2ad7a75e8ca947284454ac8bf5a856607630ef1d9adb48c0e8f" Oct 31 03:57:27.778982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:27.778922 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37213b741452d2ad7a75e8ca947284454ac8bf5a856607630ef1d9adb48c0e8f"} err="failed to get container status \"37213b741452d2ad7a75e8ca947284454ac8bf5a856607630ef1d9adb48c0e8f\": rpc error: code = NotFound desc = could not find container \"37213b741452d2ad7a75e8ca947284454ac8bf5a856607630ef1d9adb48c0e8f\": container with ID starting with 37213b741452d2ad7a75e8ca947284454ac8bf5a856607630ef1d9adb48c0e8f not found: ID does not exist" Oct 31 03:57:27.862887 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:27.862797 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qjhp\" (UniqueName: \"kubernetes.io/projected/e9f75494-3ba5-460d-a96d-9ab2f4ffa395-kube-api-access-9qjhp\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:27.862887 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:27.862826 5490 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9f75494-3ba5-460d-a96d-9ab2f4ffa395-plugins-config-volume\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:27.960418 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:27.960379 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:28.086257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:28.086215 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs"] Oct 31 03:57:28.089271 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:28.089245 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-090ebd-6d4dddff7-x6vvs"] Oct 31 03:57:28.813176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:28.813130 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:29.011409 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.011358 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" containerID="cri-o://6bef889339eba32ec308250696841174118b5d3a848c44c7fec7a22e93bba370" gracePeriod=45 Oct 31 03:57:29.269052 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.269023 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:57:29.379968 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.379871 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-dshm\") pod \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " Oct 31 03:57:29.380764 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.379967 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eca1025f-f016-4b3b-b32f-7ec9c69771b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/eca1025f-f016-4b3b-b32f-7ec9c69771b1-tls-certs\") pod \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " Oct 31 03:57:29.380764 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.380081 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eca1025f-f016-4b3b-b32f-7ec9c69771b1-kube-api-access-m5vxf\" (UniqueName: \"kubernetes.io/projected/eca1025f-f016-4b3b-b32f-7ec9c69771b1-kube-api-access-m5vxf\") pod \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " Oct 31 03:57:29.380764 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.380113 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-tmp-dir\") pod \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " Oct 31 03:57:29.380764 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.380285 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-model-cache\") pod \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " Oct 31 03:57:29.380764 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.380461 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-tmp-dir" pod "eca1025f-f016-4b3b-b32f-7ec9c69771b1" (UID: "eca1025f-f016-4b3b-b32f-7ec9c69771b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:29.381241 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.380847 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-home" pod "eca1025f-f016-4b3b-b32f-7ec9c69771b1" (UID: "eca1025f-f016-4b3b-b32f-7ec9c69771b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:29.381241 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.381230 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-model-cache" pod "eca1025f-f016-4b3b-b32f-7ec9c69771b1" (UID: "eca1025f-f016-4b3b-b32f-7ec9c69771b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:29.381364 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.381274 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-home\" (UniqueName: \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-home\") pod \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\" (UID: \"eca1025f-f016-4b3b-b32f-7ec9c69771b1\") " Oct 31 03:57:29.381847 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.381783 5490 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-tmp-dir\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.381847 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.381809 5490 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-model-cache\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.381847 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.381823 5490 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-home\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.382967 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.382933 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca1025f-f016-4b3b-b32f-7ec9c69771b1-kube-api-access-m5vxf" pod "eca1025f-f016-4b3b-b32f-7ec9c69771b1" (UID: "eca1025f-f016-4b3b-b32f-7ec9c69771b1"). InnerVolumeSpecName "kube-api-access-m5vxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:29.383082 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.382963 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-dshm" pod "eca1025f-f016-4b3b-b32f-7ec9c69771b1" (UID: "eca1025f-f016-4b3b-b32f-7ec9c69771b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:57:29.383082 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.383012 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eca1025f-f016-4b3b-b32f-7ec9c69771b1-tls-certs" pod "eca1025f-f016-4b3b-b32f-7ec9c69771b1" (UID: "eca1025f-f016-4b3b-b32f-7ec9c69771b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:57:29.483070 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.483034 5490 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eca1025f-f016-4b3b-b32f-7ec9c69771b1-dshm\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.483070 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.483064 5490 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eca1025f-f016-4b3b-b32f-7ec9c69771b1-tls-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.483070 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.483078 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5vxf\" (UniqueName: \"kubernetes.io/projected/eca1025f-f016-4b3b-b32f-7ec9c69771b1-kube-api-access-m5vxf\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:29.506591 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.506555 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:29.577983 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.577935 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f75494-3ba5-460d-a96d-9ab2f4ffa395" path="/var/lib/kubelet/pods/e9f75494-3ba5-460d-a96d-9ab2f4ffa395/volumes" Oct 31 03:57:29.625160 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.625119 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" podUID="4cf50264-0d76-46ca-b2cb-998ae3bbeb1b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:57:29.777246 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.777208 5490 generic.go:356] "Generic (PLEG): container finished" podID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerID="6bef889339eba32ec308250696841174118b5d3a848c44c7fec7a22e93bba370" exitCode=0 Oct 31 03:57:29.777426 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.777260 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" event={"ID":"eca1025f-f016-4b3b-b32f-7ec9c69771b1","Type":"ContainerDied","Data":"6bef889339eba32ec308250696841174118b5d3a848c44c7fec7a22e93bba370"} Oct 31 03:57:29.777426 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.777281 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" Oct 31 03:57:29.777426 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.777314 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc" event={"ID":"eca1025f-f016-4b3b-b32f-7ec9c69771b1","Type":"ContainerDied","Data":"a7e0934b3a09de8dd3249ee00474637b93aec23c95364c852dbb7bc00cf416a1"} Oct 31 03:57:29.777426 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.777334 5490 scope.go:122] "RemoveContainer" containerID="6bef889339eba32ec308250696841174118b5d3a848c44c7fec7a22e93bba370" Oct 31 03:57:29.788878 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.788854 5490 scope.go:122] "RemoveContainer" containerID="6bef889339eba32ec308250696841174118b5d3a848c44c7fec7a22e93bba370" Oct 31 03:57:29.789185 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:57:29.789164 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bef889339eba32ec308250696841174118b5d3a848c44c7fec7a22e93bba370\": container with ID starting with 6bef889339eba32ec308250696841174118b5d3a848c44c7fec7a22e93bba370 not found: ID does not exist" containerID="6bef889339eba32ec308250696841174118b5d3a848c44c7fec7a22e93bba370" Oct 31 03:57:29.789346 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.789194 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bef889339eba32ec308250696841174118b5d3a848c44c7fec7a22e93bba370"} err="failed to get container status \"6bef889339eba32ec308250696841174118b5d3a848c44c7fec7a22e93bba370\": rpc error: code = NotFound desc = could not find container \"6bef889339eba32ec308250696841174118b5d3a848c44c7fec7a22e93bba370\": container with ID starting with 6bef889339eba32ec308250696841174118b5d3a848c44c7fec7a22e93bba370 not found: ID does not exist" Oct 31 03:57:29.797827 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.797793 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc"] Oct 31 03:57:29.800637 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:29.800610 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-f20dc4/premium-w2-f20dc4-kserve-7c8544b957-dz7lc"] Oct 31 03:57:30.201592 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:30.201557 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:30.875318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:30.875280 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:31.056000 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:31.055954 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:31.211293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:31.211245 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:31.564365 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:31.564277 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:31.575101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:31.575065 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" path="/var/lib/kubelet/pods/eca1025f-f016-4b3b-b32f-7ec9c69771b1/volumes" Oct 31 03:57:31.766903 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:31.766865 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:32.558868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:32.558830 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:33.201501 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:33.201447 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:33.995602 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:33.995567 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:34.770311 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.770263 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:34.817579 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.817544 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj"] Oct 31 03:57:34.818165 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818142 5490 cpu_manager.go:414] "containerMap: removing container" podUID="e9f75494-3ba5-460d-a96d-9ab2f4ffa395" containerName="payload-processing" Oct 31 03:57:34.818165 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818162 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f75494-3ba5-460d-a96d-9ab2f4ffa395" containerName="payload-processing" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818174 5490 cpu_manager.go:414] "containerMap: removing container" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818179 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818188 5490 cpu_manager.go:414] "containerMap: removing container" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818194 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818209 5490 cpu_manager.go:414] "containerMap: removing container" podUID="669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0" containerName="maas-api" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818216 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0" containerName="maas-api" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818232 5490 cpu_manager.go:414] "containerMap: removing container" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818239 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818251 5490 cpu_manager.go:414] "containerMap: removing container" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818258 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818279 5490 cpu_manager.go:414] "containerMap: removing container" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818285 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818292 5490 cpu_manager.go:414] "containerMap: removing container" podUID="b0b3f9b0-9f95-4556-87a2-0d7473db75f5" containerName="istio-proxy" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818296 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b3f9b0-9f95-4556-87a2-0d7473db75f5" containerName="istio-proxy" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818304 5490 cpu_manager.go:414] "containerMap: removing container" podUID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerName="istio-proxy" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818308 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerName="istio-proxy" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818318 5490 cpu_manager.go:414] "containerMap: removing container" podUID="ceb68ad8-3da3-4469-9207-cb473d1dad9e" containerName="payload-processing" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818322 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceb68ad8-3da3-4469-9207-cb473d1dad9e" containerName="payload-processing" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818329 5490 cpu_manager.go:414] "containerMap: removing container" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818334 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818341 5490 cpu_manager.go:414] "containerMap: removing container" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818345 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818353 5490 cpu_manager.go:414] "containerMap: removing container" podUID="1ce35023-5f08-4356-8e09-4328952c6040" containerName="istio-proxy" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818357 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce35023-5f08-4356-8e09-4328952c6040" containerName="istio-proxy" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818363 5490 cpu_manager.go:414] "containerMap: removing container" podUID="a1e68997-6d71-4e7e-a80c-ca5c29aa225e" containerName="istio-proxy" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818368 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e68997-6d71-4e7e-a80c-ca5c29aa225e" containerName="istio-proxy" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818381 5490 cpu_manager.go:414] "containerMap: removing container" podUID="5d9ae28c-045f-456f-bbd7-8105b6e0077d" containerName="payload-pre-processing" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818386 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9ae28c-045f-456f-bbd7-8105b6e0077d" containerName="payload-pre-processing" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818396 5490 cpu_manager.go:414] "containerMap: removing container" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" Oct 31 03:57:34.818382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818401 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818414 5490 cpu_manager.go:414] "containerMap: removing container" podUID="c809aea3-9195-4f90-9ffb-ca153af1a5be" containerName="maas-api" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818419 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c809aea3-9195-4f90-9ffb-ca153af1a5be" containerName="maas-api" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818428 5490 cpu_manager.go:414] "containerMap: removing container" podUID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerName="istio-proxy" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818432 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerName="istio-proxy" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818438 5490 cpu_manager.go:414] "containerMap: removing container" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818442 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818449 5490 cpu_manager.go:414] "containerMap: removing container" podUID="d2a15588-0492-4ccc-96cc-ab09e1e1402d" containerName="maas-api" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818453 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2a15588-0492-4ccc-96cc-ab09e1e1402d" containerName="maas-api" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818459 5490 cpu_manager.go:414] "containerMap: removing container" podUID="288a58cd-9286-422b-bff8-15da47bc1a24" containerName="maas-api" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818465 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288a58cd-9286-422b-bff8-15da47bc1a24" containerName="maas-api" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818481 5490 cpu_manager.go:414] "containerMap: removing container" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818486 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818494 5490 cpu_manager.go:414] "containerMap: removing container" podUID="ca122c09-e327-4371-9879-0870b37db17f" containerName="istio-proxy" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818499 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca122c09-e327-4371-9879-0870b37db17f" containerName="istio-proxy" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818618 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61881b1-0191-43d7-9ac2-c060af4e8f6c" containerName="main" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818633 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca122c09-e327-4371-9879-0870b37db17f" containerName="istio-proxy" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818642 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f75494-3ba5-460d-a96d-9ab2f4ffa395" containerName="payload-processing" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818656 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c2b41fc-ff94-4a75-affd-81795799528d" containerName="main" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818667 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b771958-a3d1-40a9-aa74-81b7b33aa1e5" containerName="main" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818692 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="669f0bd3-7a5b-4e49-97d1-ca6fe5c241d0" containerName="maas-api" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818703 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f89ed71-a623-40af-a5b1-5f6333235c14" containerName="main" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818710 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef5a4fe-922e-48f2-a273-bc8c3b3f0f66" containerName="main" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818718 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e4007d-c749-49c9-b6b2-301e57a795b3" containerName="main" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818727 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b3f9b0-9f95-4556-87a2-0d7473db75f5" containerName="istio-proxy" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818737 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerName="istio-proxy" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818745 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="f91c38ef-6298-4315-9a8f-bedd67fcdd1c" containerName="istio-proxy" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818753 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceb68ad8-3da3-4469-9207-cb473d1dad9e" containerName="payload-processing" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818758 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818763 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="eca1025f-f016-4b3b-b32f-7ec9c69771b1" containerName="main" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818768 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e68997-6d71-4e7e-a80c-ca5c29aa225e" containerName="istio-proxy" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818775 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9ae28c-045f-456f-bbd7-8105b6e0077d" containerName="payload-pre-processing" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818780 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c3288c4-4250-4046-921b-4382c4d3ca93" containerName="main" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818787 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="c809aea3-9195-4f90-9ffb-ca153af1a5be" containerName="maas-api" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818794 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbec9363-80a5-497b-8dd8-45b9fba41328" containerName="istio-proxy" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818800 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0596bd0-518a-45ab-8cc1-8ba75268ef6a" containerName="istio-proxy" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818806 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2a15588-0492-4ccc-96cc-ab09e1e1402d" containerName="maas-api" Oct 31 03:57:34.820101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.818815 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="288a58cd-9286-422b-bff8-15da47bc1a24" containerName="maas-api" Oct 31 03:57:34.822445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.822425 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:34.834115 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.834081 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-34cb76-openshift-default-dockercfg-qd8ts\"" Oct 31 03:57:34.861395 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.861273 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj"] Oct 31 03:57:34.920979 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.920927 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:34.970740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.964214 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-envoy\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:34.970740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.964371 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-credential-socket\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:34.970740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.964438 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-podinfo\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:34.970740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.964528 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istiod-ca-cert\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:34.970740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.964579 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-token\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:34.970740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.964690 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdfpx\" (UniqueName: \"kubernetes.io/projected/b6b9fc40-2379-498d-84f5-ce5550fd76ae-kube-api-access-zdfpx\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:34.970740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.964723 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-data\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:34.970740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.964758 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-workload-socket\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:34.970740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:34.964815 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-workload-certs\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:35.071750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.065920 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-envoy\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:35.071750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.065996 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-credential-socket\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:35.071750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.066051 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-podinfo\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:35.071750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.066081 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istiod-ca-cert\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:35.071750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.066118 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-token\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:35.071750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.066484 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdfpx\" (UniqueName: \"kubernetes.io/projected/b6b9fc40-2379-498d-84f5-ce5550fd76ae-kube-api-access-zdfpx\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:35.071750 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.070198 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-envoy\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:35.072923 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.072895 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-credential-socket\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:35.096703 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.073722 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istiod-ca-cert\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:35.097332 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.084560 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-data\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:35.097585 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.097554 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-workload-socket\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:35.109231 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.086825 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-podinfo\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:35.109574 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.098105 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-workload-socket\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:35.109786 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.101306 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:35.110167 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.084945 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-data\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:35.110315 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.107955 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-workload-certs\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:35.110558 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.108294 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-workload-certs\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:35.112427 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.112408 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-token\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:35.117304 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.115319 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdfpx\" (UniqueName: \"kubernetes.io/projected/b6b9fc40-2379-498d-84f5-ce5550fd76ae-kube-api-access-zdfpx\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:35.192719 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.189444 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:35.277829 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.277361 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:35.509914 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.509870 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj"] Oct 31 03:57:35.511065 ip-10-0-142-76 kubenswrapper[5490]: W1031 03:57:35.511033 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b9fc40_2379_498d_84f5_ce5550fd76ae.slice/crio-1a80b56d6f0f5a85f4dbb55fe5cc6791df755429752a38fddb3a1fd226a98254 WatchSource:0}: Error finding container 1a80b56d6f0f5a85f4dbb55fe5cc6791df755429752a38fddb3a1fd226a98254: Status 404 returned error can't find the container with id 1a80b56d6f0f5a85f4dbb55fe5cc6791df755429752a38fddb3a1fd226a98254 Oct 31 03:57:35.514143 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.514108 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:57:35.514251 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.514178 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:57:35.514251 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.514204 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 03:57:35.822338 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.822240 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" event={"ID":"b6b9fc40-2379-498d-84f5-ce5550fd76ae","Type":"ContainerStarted","Data":"73c2237fb7b9b625272b3c052a08a9cadcaefe16179dc32b50a020dc070f2855"} Oct 31 03:57:35.822338 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.822328 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" event={"ID":"b6b9fc40-2379-498d-84f5-ce5550fd76ae","Type":"ContainerStarted","Data":"1a80b56d6f0f5a85f4dbb55fe5cc6791df755429752a38fddb3a1fd226a98254"} Oct 31 03:57:35.861280 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:35.861221 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" podStartSLOduration=1.861204171 podStartE2EDuration="1.861204171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:57:35.858269967 +0000 UTC m=+2490.392964923" watchObservedRunningTime="2026-10-31 03:57:35.861204171 +0000 UTC m=+2490.395899125" Oct 31 03:57:36.189766 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:36.189719 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:36.191611 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:36.191583 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 31 03:57:36.191784 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:36.191653 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" podUID="b6b9fc40-2379-498d-84f5-ce5550fd76ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 31 03:57:36.365466 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:36.365427 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:37.190408 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:37.190373 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 31 03:57:37.190851 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:37.190443 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" podUID="b6b9fc40-2379-498d-84f5-ce5550fd76ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 31 03:57:37.446813 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:37.446705 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:38.190143 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:38.190102 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 31 03:57:38.190404 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:38.190376 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" podUID="b6b9fc40-2379-498d-84f5-ce5550fd76ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 31 03:57:38.278465 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:38.278412 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:38.705934 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:38.705891 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:39.190905 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:39.190869 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 31 03:57:39.191098 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:39.190944 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" podUID="b6b9fc40-2379-498d-84f5-ce5550fd76ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 31 03:57:39.570436 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:39.570352 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:39.624554 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:39.624516 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" podUID="4cf50264-0d76-46ca-b2cb-998ae3bbeb1b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:57:40.190326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:40.190292 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 31 03:57:40.190519 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:40.190346 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" podUID="b6b9fc40-2379-498d-84f5-ce5550fd76ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 31 03:57:40.357757 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:40.353258 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:41.120618 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:41.120577 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:41.190063 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:41.190022 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 31 03:57:41.190223 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:41.190093 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" podUID="b6b9fc40-2379-498d-84f5-ce5550fd76ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 31 03:57:41.718959 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:41.718917 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:41.862122 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:41.862078 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:42.029166 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:42.029073 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:42.169524 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:42.169481 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:42.195467 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:42.195437 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:42.195932 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:42.195916 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:42.196780 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:42.196759 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:57:42.495129 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:42.495090 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:42.809352 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:42.809240 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:43.689315 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:43.689281 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:44.593255 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:44.593184 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" Oct 31 03:57:44.652466 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:44.652425 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cf50264-0d76-46ca-b2cb-998ae3bbeb1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf50264-0d76-46ca-b2cb-998ae3bbeb1b-plugins-config-volume\") pod \"4cf50264-0d76-46ca-b2cb-998ae3bbeb1b\" (UID: \"4cf50264-0d76-46ca-b2cb-998ae3bbeb1b\") " Oct 31 03:57:44.652660 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:44.652596 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf50264-0d76-46ca-b2cb-998ae3bbeb1b-kube-api-access-7ftq2\" (UniqueName: \"kubernetes.io/projected/4cf50264-0d76-46ca-b2cb-998ae3bbeb1b-kube-api-access-7ftq2\") pod \"4cf50264-0d76-46ca-b2cb-998ae3bbeb1b\" (UID: \"4cf50264-0d76-46ca-b2cb-998ae3bbeb1b\") " Oct 31 03:57:44.652921 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:44.652896 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf50264-0d76-46ca-b2cb-998ae3bbeb1b-plugins-config-volume" pod "4cf50264-0d76-46ca-b2cb-998ae3bbeb1b" (UID: "4cf50264-0d76-46ca-b2cb-998ae3bbeb1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:57:44.653129 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:44.653100 5490 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf50264-0d76-46ca-b2cb-998ae3bbeb1b-plugins-config-volume\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:44.654956 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:44.654926 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf50264-0d76-46ca-b2cb-998ae3bbeb1b-kube-api-access-7ftq2" pod "4cf50264-0d76-46ca-b2cb-998ae3bbeb1b" (UID: "4cf50264-0d76-46ca-b2cb-998ae3bbeb1b"). InnerVolumeSpecName "kube-api-access-7ftq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:57:44.672452 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:44.672406 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:44.754454 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:44.754404 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ftq2\" (UniqueName: \"kubernetes.io/projected/4cf50264-0d76-46ca-b2cb-998ae3bbeb1b-kube-api-access-7ftq2\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:57:44.928590 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:44.928553 5490 generic.go:356] "Generic (PLEG): container finished" podID="4cf50264-0d76-46ca-b2cb-998ae3bbeb1b" containerID="9a8d03166a4c32dde0efd5baaefb8adde72ac0c51ba156399e116bc5f90a3b3b" exitCode=0 Oct 31 03:57:44.928809 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:44.928605 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" Oct 31 03:57:44.928809 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:44.928643 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" event={"ID":"4cf50264-0d76-46ca-b2cb-998ae3bbeb1b","Type":"ContainerDied","Data":"9a8d03166a4c32dde0efd5baaefb8adde72ac0c51ba156399e116bc5f90a3b3b"} Oct 31 03:57:44.928809 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:44.928735 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf" event={"ID":"4cf50264-0d76-46ca-b2cb-998ae3bbeb1b","Type":"ContainerDied","Data":"1f7d68da13054c6673e5381780b144fe7d75d2b8afd3e047df14d44337dbfb16"} Oct 31 03:57:44.928809 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:44.928759 5490 scope.go:122] "RemoveContainer" containerID="9a8d03166a4c32dde0efd5baaefb8adde72ac0c51ba156399e116bc5f90a3b3b" Oct 31 03:57:44.939717 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:44.939690 5490 scope.go:122] "RemoveContainer" containerID="9a8d03166a4c32dde0efd5baaefb8adde72ac0c51ba156399e116bc5f90a3b3b" Oct 31 03:57:44.940061 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:57:44.940039 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a8d03166a4c32dde0efd5baaefb8adde72ac0c51ba156399e116bc5f90a3b3b\": container with ID starting with 9a8d03166a4c32dde0efd5baaefb8adde72ac0c51ba156399e116bc5f90a3b3b not found: ID does not exist" containerID="9a8d03166a4c32dde0efd5baaefb8adde72ac0c51ba156399e116bc5f90a3b3b" Oct 31 03:57:44.940124 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:44.940071 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a8d03166a4c32dde0efd5baaefb8adde72ac0c51ba156399e116bc5f90a3b3b"} err="failed to get container status \"9a8d03166a4c32dde0efd5baaefb8adde72ac0c51ba156399e116bc5f90a3b3b\": rpc error: code = NotFound desc = could not find container \"9a8d03166a4c32dde0efd5baaefb8adde72ac0c51ba156399e116bc5f90a3b3b\": container with ID starting with 9a8d03166a4c32dde0efd5baaefb8adde72ac0c51ba156399e116bc5f90a3b3b not found: ID does not exist" Oct 31 03:57:44.954098 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:44.954067 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf"] Oct 31 03:57:44.968310 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:44.968271 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f20dc4-6d7b555ffc-h8mtf"] Oct 31 03:57:45.535667 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:45.535215 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:45.576130 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:45.576095 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cf50264-0d76-46ca-b2cb-998ae3bbeb1b" path="/var/lib/kubelet/pods/4cf50264-0d76-46ca-b2cb-998ae3bbeb1b/volumes" Oct 31 03:57:45.951624 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:45.951567 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:46.198710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:46.198652 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:47.001437 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:47.001396 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:47.127834 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:57:47.127800 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:57:47.801373 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:47.801335 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:48.699525 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:48.699487 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:49.284489 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:49.284439 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:49.427448 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:49.427401 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:49.816590 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:49.816552 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:50.210252 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:50.210205 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:50.968257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:50.968211 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:51.096405 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:51.096370 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:51.706957 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:51.706917 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:52.314459 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:52.314416 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:52.443952 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:52.443910 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:53.201511 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:53.201472 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:54.009830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:54.009789 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:55.089071 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:55.089025 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:55.815018 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:55.814986 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:56.501019 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:56.500977 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:57.171758 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:57.171718 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:57.894928 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:57.894895 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:58.492209 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:58.492166 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:58.624218 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:58.624182 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:59.097246 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:59.097206 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:57:59.692027 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:57:59.691989 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:00.481096 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:00.481058 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:01.197235 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:01.197194 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:02.092607 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:02.092561 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:02.702168 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:02.702115 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:03.388782 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:03.388739 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:03.991787 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:03.991743 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:04.592857 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:04.592814 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:05.197654 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:05.197618 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:05.797018 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:05.796904 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:06.505405 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:06.505311 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:07.101004 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:07.100964 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:07.887559 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:07.887476 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:08.548147 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:08.548057 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:09.184987 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:09.184951 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:09.784627 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:09.784584 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:10.378031 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:10.377987 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:10.995953 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:10.995914 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:11.588415 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:11.588374 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:12.179275 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:12.179234 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:12.791035 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:12.790978 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:13.379552 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:13.379516 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:13.986452 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:13.986406 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:14.581760 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:14.581721 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:15.210655 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:15.210618 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:15.948585 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:15.948491 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:16.744498 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:16.744463 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:17.129167 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:58:17.129072 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:58:17.402618 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:17.402516 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:18.305113 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:18.305087 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ttmkb_400b676a-a3a0-459d-9da6-8d30b2d3b497/csi-driver/0.log" Oct 31 03:58:18.393054 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:18.392997 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:18.522338 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:18.522294 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:18.999933 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:18.999895 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:19.686359 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:19.686318 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:20.281382 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:20.281305 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:20.896502 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:20.896275 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:21.494793 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:21.494746 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:22.150066 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:22.150029 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:22.860888 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:22.860846 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:23.690017 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:23.689979 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:24.298934 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:24.298893 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:24.897890 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:24.897853 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:25.500961 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:25.500913 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:26.230535 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:26.230488 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:26.895931 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:26.895862 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:27.504293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:27.504252 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:28.101276 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:28.101234 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:28.799897 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:28.799860 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:29.396838 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:29.396796 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:30.136896 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:30.136861 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:30.823858 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:30.823816 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:31.617526 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:31.617496 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:32.307922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:32.307885 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:33.009176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:33.009138 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:33.787455 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:33.787418 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:34.496463 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:34.496427 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:35.095470 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:35.095430 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:35.544176 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:35.544149 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ttmkb_400b676a-a3a0-459d-9da6-8d30b2d3b497/csi-driver/0.log" Oct 31 03:58:35.805475 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:35.805362 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:36.401261 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:36.401220 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:37.005117 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:37.005076 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:37.599154 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:37.598456 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:38.210385 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:38.210340 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:38.808837 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:38.808799 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:39.599367 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:39.599327 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:40.299160 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:40.299118 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:41.012839 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:41.012803 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:41.797852 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:41.797807 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:42.415392 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:42.415340 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:43.077499 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:43.077457 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:43.701513 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:43.701452 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:44.393048 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:44.393010 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:45.319615 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:45.319574 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:45.986020 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:45.985978 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:46.601198 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:46.601157 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:47.127905 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:58:47.127862 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:58:47.280557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:47.280515 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:47.895710 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:47.895641 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:48.501943 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:48.501903 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:49.181490 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:49.181450 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:49.789771 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:49.789730 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:50.739448 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:50.739411 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:51.037018 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:51.036931 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:51.300235 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:51.300145 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:51.975935 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:51.975903 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:52.138522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:52.138481 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:52.675804 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:52.675760 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:53.398355 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:53.396918 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:54.091278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:54.091200 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:54.894445 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:54.894354 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:55.829050 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:55.829009 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:56.761026 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:56.760987 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:56.944802 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:56.944759 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:57.671279 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:57.671241 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:58:58.470376 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:58:58.470338 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:00.228224 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:00.228166 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:00.356495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:00.356455 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:00.541941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:00.541666 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:01.395235 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:01.395196 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:02.001305 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:02.001247 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:02.667686 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:02.667631 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:03.282922 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:03.282879 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:03.893268 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:03.893222 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:04.485173 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:04.485109 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:05.096759 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:05.096720 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:05.798722 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:05.798665 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:06.650347 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:06.650311 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:07.146248 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:07.146205 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5"] Oct 31 03:59:07.157269 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:07.157231 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq"] Oct 31 03:59:07.264068 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:07.263983 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:07.394038 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:07.394002 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:07.958488 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:07.958447 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:08.378527 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:08.378428 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb"] Oct 31 03:59:08.378794 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:08.378741 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" podUID="7d55416a-52be-41b0-aef9-7709346f1fdd" containerName="maas-api" containerID="cri-o://54f575127787e893c81f50bae1cffc63850e0622cd625e4dd188a748b0a6066d" gracePeriod=30 Oct 31 03:59:08.462515 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:08.462479 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:08.595336 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:08.591522 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:08.689294 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:08.689258 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" Oct 31 03:59:08.706346 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:08.706300 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d55416a-52be-41b0-aef9-7709346f1fdd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d55416a-52be-41b0-aef9-7709346f1fdd-openshift-service-ca\") pod \"7d55416a-52be-41b0-aef9-7709346f1fdd\" (UID: \"7d55416a-52be-41b0-aef9-7709346f1fdd\") " Oct 31 03:59:08.706562 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:08.706412 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d55416a-52be-41b0-aef9-7709346f1fdd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d55416a-52be-41b0-aef9-7709346f1fdd-maas-api-tls\") pod \"7d55416a-52be-41b0-aef9-7709346f1fdd\" (UID: \"7d55416a-52be-41b0-aef9-7709346f1fdd\") " Oct 31 03:59:08.706562 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:08.706490 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d55416a-52be-41b0-aef9-7709346f1fdd-kube-api-access-dlctl\" (UniqueName: \"kubernetes.io/projected/7d55416a-52be-41b0-aef9-7709346f1fdd-kube-api-access-dlctl\") pod \"7d55416a-52be-41b0-aef9-7709346f1fdd\" (UID: \"7d55416a-52be-41b0-aef9-7709346f1fdd\") " Oct 31 03:59:08.706562 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:08.706530 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d55416a-52be-41b0-aef9-7709346f1fdd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d55416a-52be-41b0-aef9-7709346f1fdd-metrics-certs\") pod \"7d55416a-52be-41b0-aef9-7709346f1fdd\" (UID: \"7d55416a-52be-41b0-aef9-7709346f1fdd\") " Oct 31 03:59:08.707746 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:08.707407 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d55416a-52be-41b0-aef9-7709346f1fdd-openshift-service-ca" pod "7d55416a-52be-41b0-aef9-7709346f1fdd" (UID: "7d55416a-52be-41b0-aef9-7709346f1fdd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:08.709554 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:08.709154 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d55416a-52be-41b0-aef9-7709346f1fdd-maas-api-tls" pod "7d55416a-52be-41b0-aef9-7709346f1fdd" (UID: "7d55416a-52be-41b0-aef9-7709346f1fdd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:08.709554 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:08.709375 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d55416a-52be-41b0-aef9-7709346f1fdd-metrics-certs" pod "7d55416a-52be-41b0-aef9-7709346f1fdd" (UID: "7d55416a-52be-41b0-aef9-7709346f1fdd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:08.710101 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:08.710076 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d55416a-52be-41b0-aef9-7709346f1fdd-kube-api-access-dlctl" pod "7d55416a-52be-41b0-aef9-7709346f1fdd" (UID: "7d55416a-52be-41b0-aef9-7709346f1fdd"). InnerVolumeSpecName "kube-api-access-dlctl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:08.721516 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:08.721424 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:08.807726 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:08.807654 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d55416a-52be-41b0-aef9-7709346f1fdd-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:08.807726 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:08.807719 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d55416a-52be-41b0-aef9-7709346f1fdd-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:08.807996 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:08.807734 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlctl\" (UniqueName: \"kubernetes.io/projected/7d55416a-52be-41b0-aef9-7709346f1fdd-kube-api-access-dlctl\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:08.807996 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:08.807749 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d55416a-52be-41b0-aef9-7709346f1fdd-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:08.857347 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:08.857311 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:09.295522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:09.295477 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:09.486778 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:09.486738 5490 generic.go:356] "Generic (PLEG): container finished" podID="7d55416a-52be-41b0-aef9-7709346f1fdd" containerID="54f575127787e893c81f50bae1cffc63850e0622cd625e4dd188a748b0a6066d" exitCode=0 Oct 31 03:59:09.486993 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:09.486789 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" Oct 31 03:59:09.486993 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:09.486790 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" event={"ID":"7d55416a-52be-41b0-aef9-7709346f1fdd","Type":"ContainerDied","Data":"54f575127787e893c81f50bae1cffc63850e0622cd625e4dd188a748b0a6066d"} Oct 31 03:59:09.486993 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:09.486910 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb" event={"ID":"7d55416a-52be-41b0-aef9-7709346f1fdd","Type":"ContainerDied","Data":"ca01093eedb903f00d9ebaec9ba2499969c11c515ae79c69fd5f799f55f6b4e6"} Oct 31 03:59:09.486993 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:09.486936 5490 scope.go:122] "RemoveContainer" containerID="54f575127787e893c81f50bae1cffc63850e0622cd625e4dd188a748b0a6066d" Oct 31 03:59:09.507779 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:09.507583 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:09.512562 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:09.512530 5490 scope.go:122] "RemoveContainer" containerID="54f575127787e893c81f50bae1cffc63850e0622cd625e4dd188a748b0a6066d" Oct 31 03:59:09.513022 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:59:09.512986 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f575127787e893c81f50bae1cffc63850e0622cd625e4dd188a748b0a6066d\": container with ID starting with 54f575127787e893c81f50bae1cffc63850e0622cd625e4dd188a748b0a6066d not found: ID does not exist" containerID="54f575127787e893c81f50bae1cffc63850e0622cd625e4dd188a748b0a6066d" Oct 31 03:59:09.513131 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:09.513030 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f575127787e893c81f50bae1cffc63850e0622cd625e4dd188a748b0a6066d"} err="failed to get container status \"54f575127787e893c81f50bae1cffc63850e0622cd625e4dd188a748b0a6066d\": rpc error: code = NotFound desc = could not find container \"54f575127787e893c81f50bae1cffc63850e0622cd625e4dd188a748b0a6066d\": container with ID starting with 54f575127787e893c81f50bae1cffc63850e0622cd625e4dd188a748b0a6066d not found: ID does not exist" Oct 31 03:59:09.530729 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:09.524614 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb"] Oct 31 03:59:09.530729 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:09.527908 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-20585c-59dcc777fc-mljxb"] Oct 31 03:59:09.575022 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:09.574932 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d55416a-52be-41b0-aef9-7709346f1fdd" path="/var/lib/kubelet/pods/7d55416a-52be-41b0-aef9-7709346f1fdd/volumes" Oct 31 03:59:10.098636 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:10.098553 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:10.771488 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:10.771450 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:11.386576 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:11.386539 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:11.991976 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:11.991932 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:12.147415 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:12.147367 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5" podUID="2cf184db-500a-41eb-8ca5-7a40b2645693" containerName="payload-processing" containerID="cri-o://1ed37b99793e18547ecfef0a6931a7ab57f9c50ce6f0b6efbb24bade256fc997" gracePeriod=25 Oct 31 03:59:12.157958 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:12.157916 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq" podUID="679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2" containerName="payload-pre-processing" containerID="cri-o://ed6d1bdf222e1a94cde8ab9102d20da712aa9cec10da51754d6641048c8c4ca8" gracePeriod=25 Oct 31 03:59:12.699182 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:12.699140 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:13.306865 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:13.306812 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:13.969544 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:13.969506 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:14.781586 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:14.781542 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:15.553693 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:15.553644 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:15.688933 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:15.688889 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:15.938549 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:15.938511 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59"] Oct 31 03:59:15.939039 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:15.938774 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" podUID="24a03b21-4c4f-422e-baa4-6e905b1b0bf5" containerName="istio-proxy" containerID="cri-o://ebbcf30bc2205d2f85a6a78f2f8b275f7f73bac58e4826c35d31601c7ee0a9e7" gracePeriod=30 Oct 31 03:59:16.026704 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:16.026632 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:16.183189 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:16.183148 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:16.866051 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:16.864909 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:17.117719 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:17.111309 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:17.146055 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:59:17.146019 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:59:17.750455 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:17.750416 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:17.975452 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:17.975403 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:18.500936 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:18.500882 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:18.907911 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:18.907873 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:19.209206 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:19.209108 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:19.621278 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:19.621152 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:20.304417 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:20.304379 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:21.138152 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.138115 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:21.207099 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.207033 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:59:21.347848 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.347816 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-credential-socket\") pod \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " Oct 31 03:59:21.348032 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.347880 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-kube-api-access-6hd5l\" (UniqueName: \"kubernetes.io/projected/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-kube-api-access-6hd5l\") pod \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " Oct 31 03:59:21.348032 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.347900 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-token\" (UniqueName: \"kubernetes.io/projected/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-token\") pod \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " Oct 31 03:59:21.348032 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.347946 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-workload-certs\") pod \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " Oct 31 03:59:21.348032 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.347971 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istiod-ca-cert\") pod \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " Oct 31 03:59:21.348032 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.347996 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-workload-socket\") pod \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " Oct 31 03:59:21.348332 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.348045 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-envoy\") pod \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " Oct 31 03:59:21.348332 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.348053 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-credential-socket" pod "24a03b21-4c4f-422e-baa4-6e905b1b0bf5" (UID: "24a03b21-4c4f-422e-baa4-6e905b1b0bf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:21.348332 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.348086 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-data\") pod \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " Oct 31 03:59:21.348332 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.348155 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-podinfo\") pod \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\" (UID: \"24a03b21-4c4f-422e-baa4-6e905b1b0bf5\") " Oct 31 03:59:21.348332 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.348299 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-workload-certs" pod "24a03b21-4c4f-422e-baa4-6e905b1b0bf5" (UID: "24a03b21-4c4f-422e-baa4-6e905b1b0bf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:21.348600 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.348341 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-workload-socket" pod "24a03b21-4c4f-422e-baa4-6e905b1b0bf5" (UID: "24a03b21-4c4f-422e-baa4-6e905b1b0bf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:21.348600 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.348432 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istiod-ca-cert" pod "24a03b21-4c4f-422e-baa4-6e905b1b0bf5" (UID: "24a03b21-4c4f-422e-baa4-6e905b1b0bf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:21.348600 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.348513 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-data" pod "24a03b21-4c4f-422e-baa4-6e905b1b0bf5" (UID: "24a03b21-4c4f-422e-baa4-6e905b1b0bf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:21.348801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.348599 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.348801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.348626 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.348801 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.348648 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.350392 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.350358 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-token" pod "24a03b21-4c4f-422e-baa4-6e905b1b0bf5" (UID: "24a03b21-4c4f-422e-baa4-6e905b1b0bf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:21.350522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.350434 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-envoy" pod "24a03b21-4c4f-422e-baa4-6e905b1b0bf5" (UID: "24a03b21-4c4f-422e-baa4-6e905b1b0bf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:21.350522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.350459 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-podinfo" pod "24a03b21-4c4f-422e-baa4-6e905b1b0bf5" (UID: "24a03b21-4c4f-422e-baa4-6e905b1b0bf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:59:21.350522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.350508 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-kube-api-access-6hd5l" pod "24a03b21-4c4f-422e-baa4-6e905b1b0bf5" (UID: "24a03b21-4c4f-422e-baa4-6e905b1b0bf5"). InnerVolumeSpecName "kube-api-access-6hd5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:21.449593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.449554 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hd5l\" (UniqueName: \"kubernetes.io/projected/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-kube-api-access-6hd5l\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.449593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.449589 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.449593 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.449602 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.449861 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.449615 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.449861 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.449628 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.449861 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.449639 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a03b21-4c4f-422e-baa4-6e905b1b0bf5-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:21.556350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.556269 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq" podUID="679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:59:21.556350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.556294 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5" podUID="2cf184db-500a-41eb-8ca5-7a40b2645693" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:59:21.589734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.581708 5490 generic.go:356] "Generic (PLEG): container finished" podID="24a03b21-4c4f-422e-baa4-6e905b1b0bf5" containerID="ebbcf30bc2205d2f85a6a78f2f8b275f7f73bac58e4826c35d31601c7ee0a9e7" exitCode=0 Oct 31 03:59:21.589734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.581914 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" Oct 31 03:59:21.589734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.582787 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" event={"ID":"24a03b21-4c4f-422e-baa4-6e905b1b0bf5","Type":"ContainerDied","Data":"ebbcf30bc2205d2f85a6a78f2f8b275f7f73bac58e4826c35d31601c7ee0a9e7"} Oct 31 03:59:21.589734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.582851 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59" event={"ID":"24a03b21-4c4f-422e-baa4-6e905b1b0bf5","Type":"ContainerDied","Data":"5507ec24be6b2f8a27902bcaf56e493e87f883e1db69744fb2b2bbf5632c0af6"} Oct 31 03:59:21.589734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.582882 5490 scope.go:122] "RemoveContainer" containerID="ebbcf30bc2205d2f85a6a78f2f8b275f7f73bac58e4826c35d31601c7ee0a9e7" Oct 31 03:59:21.595231 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.595211 5490 scope.go:122] "RemoveContainer" containerID="770d923a22550d4a90ab065ee3eb02df2ca0b1192f8fe0fbe642e55baff56e6b" Oct 31 03:59:21.604947 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.604917 5490 scope.go:122] "RemoveContainer" containerID="ebbcf30bc2205d2f85a6a78f2f8b275f7f73bac58e4826c35d31601c7ee0a9e7" Oct 31 03:59:21.605424 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:59:21.605385 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebbcf30bc2205d2f85a6a78f2f8b275f7f73bac58e4826c35d31601c7ee0a9e7\": container with ID starting with ebbcf30bc2205d2f85a6a78f2f8b275f7f73bac58e4826c35d31601c7ee0a9e7 not found: ID does not exist" containerID="ebbcf30bc2205d2f85a6a78f2f8b275f7f73bac58e4826c35d31601c7ee0a9e7" Oct 31 03:59:21.606062 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.605424 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebbcf30bc2205d2f85a6a78f2f8b275f7f73bac58e4826c35d31601c7ee0a9e7"} err="failed to get container status \"ebbcf30bc2205d2f85a6a78f2f8b275f7f73bac58e4826c35d31601c7ee0a9e7\": rpc error: code = NotFound desc = could not find container \"ebbcf30bc2205d2f85a6a78f2f8b275f7f73bac58e4826c35d31601c7ee0a9e7\": container with ID starting with ebbcf30bc2205d2f85a6a78f2f8b275f7f73bac58e4826c35d31601c7ee0a9e7 not found: ID does not exist" Oct 31 03:59:21.606062 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.605452 5490 scope.go:122] "RemoveContainer" containerID="770d923a22550d4a90ab065ee3eb02df2ca0b1192f8fe0fbe642e55baff56e6b" Oct 31 03:59:21.606062 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:59:21.605986 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"770d923a22550d4a90ab065ee3eb02df2ca0b1192f8fe0fbe642e55baff56e6b\": container with ID starting with 770d923a22550d4a90ab065ee3eb02df2ca0b1192f8fe0fbe642e55baff56e6b not found: ID does not exist" containerID="770d923a22550d4a90ab065ee3eb02df2ca0b1192f8fe0fbe642e55baff56e6b" Oct 31 03:59:21.606062 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.606017 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770d923a22550d4a90ab065ee3eb02df2ca0b1192f8fe0fbe642e55baff56e6b"} err="failed to get container status \"770d923a22550d4a90ab065ee3eb02df2ca0b1192f8fe0fbe642e55baff56e6b\": rpc error: code = NotFound desc = could not find container \"770d923a22550d4a90ab065ee3eb02df2ca0b1192f8fe0fbe642e55baff56e6b\": container with ID starting with 770d923a22550d4a90ab065ee3eb02df2ca0b1192f8fe0fbe642e55baff56e6b not found: ID does not exist" Oct 31 03:59:21.607880 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.607860 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59"] Oct 31 03:59:21.610698 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:21.610666 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-svd59"] Oct 31 03:59:22.011429 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:22.011386 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:22.800318 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:22.800218 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:23.479830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:23.479796 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:23.574245 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:23.574206 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24a03b21-4c4f-422e-baa4-6e905b1b0bf5" path="/var/lib/kubelet/pods/24a03b21-4c4f-422e-baa4-6e905b1b0bf5/volumes" Oct 31 03:59:24.217521 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:24.217484 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:24.789211 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:24.789181 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:25.389631 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:25.389580 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:25.989353 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:25.989310 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:26.606140 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:26.606085 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:26.741778 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:26.741740 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:26.863190 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:26.862375 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:27.276069 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:27.276033 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:27.892937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:27.892896 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:28.075257 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:28.075212 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw"] Oct 31 03:59:28.229830 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:28.229795 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:28.401503 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:28.401467 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:28.702417 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:28.702378 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:29.289112 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:29.289071 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:29.899802 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:29.899765 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:30.404971 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.404934 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl"] Oct 31 03:59:30.405655 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.405181 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" podUID="76c026b4-65c7-4d57-9945-d4ebfb66ca70" containerName="maas-api" containerID="cri-o://b785a251429aa9dc9811da6efd751dd1824b4436e966c96790aa9465afabad00" gracePeriod=30 Oct 31 03:59:30.410159 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.410113 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:30.544384 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.544345 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:30.658630 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.658534 5490 generic.go:356] "Generic (PLEG): container finished" podID="76c026b4-65c7-4d57-9945-d4ebfb66ca70" containerID="b785a251429aa9dc9811da6efd751dd1824b4436e966c96790aa9465afabad00" exitCode=0 Oct 31 03:59:30.658870 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.658669 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" event={"ID":"76c026b4-65c7-4d57-9945-d4ebfb66ca70","Type":"ContainerDied","Data":"b785a251429aa9dc9811da6efd751dd1824b4436e966c96790aa9465afabad00"} Oct 31 03:59:30.672387 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.672350 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:30.699592 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.699567 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" Oct 31 03:59:30.849075 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.849037 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:30.855293 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.855262 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76c026b4-65c7-4d57-9945-d4ebfb66ca70-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c026b4-65c7-4d57-9945-d4ebfb66ca70-openshift-service-ca\") pod \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\" (UID: \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\") " Oct 31 03:59:30.855482 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.855358 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c026b4-65c7-4d57-9945-d4ebfb66ca70-kube-api-access-842q5\" (UniqueName: \"kubernetes.io/projected/76c026b4-65c7-4d57-9945-d4ebfb66ca70-kube-api-access-842q5\") pod \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\" (UID: \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\") " Oct 31 03:59:30.855482 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.855393 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76c026b4-65c7-4d57-9945-d4ebfb66ca70-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76c026b4-65c7-4d57-9945-d4ebfb66ca70-maas-api-tls\") pod \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\" (UID: \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\") " Oct 31 03:59:30.855643 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.855558 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76c026b4-65c7-4d57-9945-d4ebfb66ca70-metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c026b4-65c7-4d57-9945-d4ebfb66ca70-metrics-certs\") pod \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\" (UID: \"76c026b4-65c7-4d57-9945-d4ebfb66ca70\") " Oct 31 03:59:30.855755 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.855646 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c026b4-65c7-4d57-9945-d4ebfb66ca70-openshift-service-ca" pod "76c026b4-65c7-4d57-9945-d4ebfb66ca70" (UID: "76c026b4-65c7-4d57-9945-d4ebfb66ca70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:30.855950 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.855917 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76c026b4-65c7-4d57-9945-d4ebfb66ca70-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:30.857576 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.857539 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c026b4-65c7-4d57-9945-d4ebfb66ca70-maas-api-tls" pod "76c026b4-65c7-4d57-9945-d4ebfb66ca70" (UID: "76c026b4-65c7-4d57-9945-d4ebfb66ca70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:30.857735 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.857670 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c026b4-65c7-4d57-9945-d4ebfb66ca70-metrics-certs" pod "76c026b4-65c7-4d57-9945-d4ebfb66ca70" (UID: "76c026b4-65c7-4d57-9945-d4ebfb66ca70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:30.857818 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.857726 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c026b4-65c7-4d57-9945-d4ebfb66ca70-kube-api-access-842q5" pod "76c026b4-65c7-4d57-9945-d4ebfb66ca70" (UID: "76c026b4-65c7-4d57-9945-d4ebfb66ca70"). InnerVolumeSpecName "kube-api-access-842q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:30.957555 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.957470 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76c026b4-65c7-4d57-9945-d4ebfb66ca70-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:30.957555 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.957514 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-842q5\" (UniqueName: \"kubernetes.io/projected/76c026b4-65c7-4d57-9945-d4ebfb66ca70-kube-api-access-842q5\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:30.957555 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.957529 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76c026b4-65c7-4d57-9945-d4ebfb66ca70-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:30.990350 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:30.990316 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:31.123908 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:31.123873 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:31.383335 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:31.383240 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:31.557609 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:31.557560 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq" podUID="679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:59:31.557609 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:31.557607 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5" podUID="2cf184db-500a-41eb-8ca5-7a40b2645693" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:59:31.666821 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:31.666789 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" Oct 31 03:59:31.667034 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:31.666791 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl" event={"ID":"76c026b4-65c7-4d57-9945-d4ebfb66ca70","Type":"ContainerDied","Data":"e2d087b176d7c27a81801a0a685d26800cdfc9d1b39b61b4c4751b7fd36c7915"} Oct 31 03:59:31.667034 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:31.666938 5490 scope.go:122] "RemoveContainer" containerID="b785a251429aa9dc9811da6efd751dd1824b4436e966c96790aa9465afabad00" Oct 31 03:59:31.690758 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:31.690721 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl"] Oct 31 03:59:31.698058 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:31.698031 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-34cb76-6d445b868b-67wfl"] Oct 31 03:59:31.750845 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:31.750807 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:32.425820 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.425788 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5" Oct 31 03:59:32.445982 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.445959 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq" Oct 31 03:59:32.574952 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.574919 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2-kube-api-access-m4d6w\" (UniqueName: \"kubernetes.io/projected/679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2-kube-api-access-m4d6w\") pod \"679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2\" (UID: \"679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2\") " Oct 31 03:59:32.575437 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.575030 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf184db-500a-41eb-8ca5-7a40b2645693-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cf184db-500a-41eb-8ca5-7a40b2645693-plugins-config-volume\") pod \"2cf184db-500a-41eb-8ca5-7a40b2645693\" (UID: \"2cf184db-500a-41eb-8ca5-7a40b2645693\") " Oct 31 03:59:32.575437 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.575061 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2-plugins-config-volume\") pod \"679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2\" (UID: \"679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2\") " Oct 31 03:59:32.575437 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.575084 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf184db-500a-41eb-8ca5-7a40b2645693-kube-api-access-7ckr4\" (UniqueName: \"kubernetes.io/projected/2cf184db-500a-41eb-8ca5-7a40b2645693-kube-api-access-7ckr4\") pod \"2cf184db-500a-41eb-8ca5-7a40b2645693\" (UID: \"2cf184db-500a-41eb-8ca5-7a40b2645693\") " Oct 31 03:59:32.575585 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.575448 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2-plugins-config-volume" pod "679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2" (UID: "679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:32.575585 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.575479 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf184db-500a-41eb-8ca5-7a40b2645693-plugins-config-volume" pod "2cf184db-500a-41eb-8ca5-7a40b2645693" (UID: "2cf184db-500a-41eb-8ca5-7a40b2645693"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:32.577193 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.577164 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf184db-500a-41eb-8ca5-7a40b2645693-kube-api-access-7ckr4" pod "2cf184db-500a-41eb-8ca5-7a40b2645693" (UID: "2cf184db-500a-41eb-8ca5-7a40b2645693"). InnerVolumeSpecName "kube-api-access-7ckr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:32.577344 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.577225 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2-kube-api-access-m4d6w" pod "679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2" (UID: "679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2"). InnerVolumeSpecName "kube-api-access-m4d6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:32.675114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.675012 5490 generic.go:356] "Generic (PLEG): container finished" podID="679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2" containerID="ed6d1bdf222e1a94cde8ab9102d20da712aa9cec10da51754d6641048c8c4ca8" exitCode=0 Oct 31 03:59:32.675114 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.675087 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq" Oct 31 03:59:32.675356 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.675105 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq" event={"ID":"679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2","Type":"ContainerDied","Data":"ed6d1bdf222e1a94cde8ab9102d20da712aa9cec10da51754d6641048c8c4ca8"} Oct 31 03:59:32.675356 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.675166 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq" event={"ID":"679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2","Type":"ContainerDied","Data":"5daec0a94d37c5e5a88956ffa41668ff18c9e24bd8bc39db874cd4d01737221f"} Oct 31 03:59:32.675356 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.675190 5490 scope.go:122] "RemoveContainer" containerID="ed6d1bdf222e1a94cde8ab9102d20da712aa9cec10da51754d6641048c8c4ca8" Oct 31 03:59:32.675977 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.675949 5490 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cf184db-500a-41eb-8ca5-7a40b2645693-plugins-config-volume\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:32.675977 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.675978 5490 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2-plugins-config-volume\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:32.676211 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.675992 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ckr4\" (UniqueName: \"kubernetes.io/projected/2cf184db-500a-41eb-8ca5-7a40b2645693-kube-api-access-7ckr4\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:32.676211 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.676006 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4d6w\" (UniqueName: \"kubernetes.io/projected/679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2-kube-api-access-m4d6w\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:32.677354 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.677327 5490 generic.go:356] "Generic (PLEG): container finished" podID="2cf184db-500a-41eb-8ca5-7a40b2645693" containerID="1ed37b99793e18547ecfef0a6931a7ab57f9c50ce6f0b6efbb24bade256fc997" exitCode=0 Oct 31 03:59:32.677467 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.677378 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5" Oct 31 03:59:32.677467 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.677420 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5" event={"ID":"2cf184db-500a-41eb-8ca5-7a40b2645693","Type":"ContainerDied","Data":"1ed37b99793e18547ecfef0a6931a7ab57f9c50ce6f0b6efbb24bade256fc997"} Oct 31 03:59:32.677557 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.677476 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5" event={"ID":"2cf184db-500a-41eb-8ca5-7a40b2645693","Type":"ContainerDied","Data":"63fcabdf2c16dab93ef2fdc1bc2c560b4209d655e6cc00858ceb0ca8bbbc54f2"} Oct 31 03:59:32.686471 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.686436 5490 scope.go:122] "RemoveContainer" containerID="ed6d1bdf222e1a94cde8ab9102d20da712aa9cec10da51754d6641048c8c4ca8" Oct 31 03:59:32.686780 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:59:32.686760 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed6d1bdf222e1a94cde8ab9102d20da712aa9cec10da51754d6641048c8c4ca8\": container with ID starting with ed6d1bdf222e1a94cde8ab9102d20da712aa9cec10da51754d6641048c8c4ca8 not found: ID does not exist" containerID="ed6d1bdf222e1a94cde8ab9102d20da712aa9cec10da51754d6641048c8c4ca8" Oct 31 03:59:32.686895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.686790 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed6d1bdf222e1a94cde8ab9102d20da712aa9cec10da51754d6641048c8c4ca8"} err="failed to get container status \"ed6d1bdf222e1a94cde8ab9102d20da712aa9cec10da51754d6641048c8c4ca8\": rpc error: code = NotFound desc = could not find container \"ed6d1bdf222e1a94cde8ab9102d20da712aa9cec10da51754d6641048c8c4ca8\": container with ID starting with ed6d1bdf222e1a94cde8ab9102d20da712aa9cec10da51754d6641048c8c4ca8 not found: ID does not exist" Oct 31 03:59:32.686895 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.686812 5490 scope.go:122] "RemoveContainer" containerID="1ed37b99793e18547ecfef0a6931a7ab57f9c50ce6f0b6efbb24bade256fc997" Oct 31 03:59:32.695579 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.695559 5490 scope.go:122] "RemoveContainer" containerID="1ed37b99793e18547ecfef0a6931a7ab57f9c50ce6f0b6efbb24bade256fc997" Oct 31 03:59:32.695913 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:59:32.695891 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed37b99793e18547ecfef0a6931a7ab57f9c50ce6f0b6efbb24bade256fc997\": container with ID starting with 1ed37b99793e18547ecfef0a6931a7ab57f9c50ce6f0b6efbb24bade256fc997 not found: ID does not exist" containerID="1ed37b99793e18547ecfef0a6931a7ab57f9c50ce6f0b6efbb24bade256fc997" Oct 31 03:59:32.695992 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.695919 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed37b99793e18547ecfef0a6931a7ab57f9c50ce6f0b6efbb24bade256fc997"} err="failed to get container status \"1ed37b99793e18547ecfef0a6931a7ab57f9c50ce6f0b6efbb24bade256fc997\": rpc error: code = NotFound desc = could not find container \"1ed37b99793e18547ecfef0a6931a7ab57f9c50ce6f0b6efbb24bade256fc997\": container with ID starting with 1ed37b99793e18547ecfef0a6931a7ab57f9c50ce6f0b6efbb24bade256fc997 not found: ID does not exist" Oct 31 03:59:32.703469 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.703363 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5"] Oct 31 03:59:32.707449 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.707403 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-20585c-54f6fcb8ff-ch4s5"] Oct 31 03:59:32.721586 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.721553 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:32.724416 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.724389 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq"] Oct 31 03:59:32.730586 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:32.730562 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-20585c-54fc4bcb7f-zdjwq"] Oct 31 03:59:33.079718 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:33.079602 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" podUID="42e53881-926d-482d-9557-a86d3ae3f9d9" containerName="payload-pre-processing" containerID="cri-o://dfe524b27adecb16940953181c7b9e812fd05a362305d00c551e3c6d3d67a10b" gracePeriod=25 Oct 31 03:59:33.429486 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:33.429453 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:33.575031 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:33.574995 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf184db-500a-41eb-8ca5-7a40b2645693" path="/var/lib/kubelet/pods/2cf184db-500a-41eb-8ca5-7a40b2645693/volumes" Oct 31 03:59:33.575610 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:33.575585 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2" path="/var/lib/kubelet/pods/679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2/volumes" Oct 31 03:59:33.576050 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:33.576030 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76c026b4-65c7-4d57-9945-d4ebfb66ca70" path="/var/lib/kubelet/pods/76c026b4-65c7-4d57-9945-d4ebfb66ca70/volumes" Oct 31 03:59:34.211850 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:34.211812 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:34.886800 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:34.885920 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:35.497867 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:35.497816 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:36.185692 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:36.185617 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:36.959634 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:36.959592 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:37.511208 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:37.511175 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:37.652117 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:37.652077 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:37.868522 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:37.868426 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:38.347058 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:38.346987 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:38.488722 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:38.488632 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:38.632274 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:38.632169 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:38.943941 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:38.943907 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj"] Oct 31 03:59:38.944146 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:38.944123 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" podUID="b6b9fc40-2379-498d-84f5-ce5550fd76ae" containerName="istio-proxy" containerID="cri-o://73c2237fb7b9b625272b3c052a08a9cadcaefe16179dc32b50a020dc070f2855" gracePeriod=30 Oct 31 03:59:39.048032 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:39.047016 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:39.158640 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:39.158600 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" podUID="42e53881-926d-482d-9557-a86d3ae3f9d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:59:39.638797 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:39.638708 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:40.425827 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:40.425790 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:40.563095 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:40.563049 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:41.218274 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:41.218234 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:42.121663 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:42.121620 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:42.198171 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:42.198135 5490 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 03:59:42.198376 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:42.198196 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" podUID="b6b9fc40-2379-498d-84f5-ce5550fd76ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 03:59:42.891979 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:42.891944 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:43.410476 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:43.410444 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:43.581191 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:43.581145 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:44.224846 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.224770 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:59:44.294859 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.294824 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-data\") pod \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " Oct 31 03:59:44.295868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.294881 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-workload-socket\") pod \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " Oct 31 03:59:44.295868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.294906 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istiod-ca-cert\") pod \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " Oct 31 03:59:44.295868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.294927 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-credential-socket\") pod \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " Oct 31 03:59:44.295868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.294974 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-token\" (UniqueName: \"kubernetes.io/projected/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-token\") pod \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " Oct 31 03:59:44.295868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.295130 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-envoy\") pod \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " Oct 31 03:59:44.295868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.295197 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-podinfo\") pod \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " Oct 31 03:59:44.295868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.295225 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-workload-certs\") pod \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " Oct 31 03:59:44.295868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.295242 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-workload-socket" pod "b6b9fc40-2379-498d-84f5-ce5550fd76ae" (UID: "b6b9fc40-2379-498d-84f5-ce5550fd76ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:44.295868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.295263 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b9fc40-2379-498d-84f5-ce5550fd76ae-kube-api-access-zdfpx\" (UniqueName: \"kubernetes.io/projected/b6b9fc40-2379-498d-84f5-ce5550fd76ae-kube-api-access-zdfpx\") pod \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\" (UID: \"b6b9fc40-2379-498d-84f5-ce5550fd76ae\") " Oct 31 03:59:44.295868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.295249 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-data" pod "b6b9fc40-2379-498d-84f5-ce5550fd76ae" (UID: "b6b9fc40-2379-498d-84f5-ce5550fd76ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:44.295868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.295419 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-credential-socket" pod "b6b9fc40-2379-498d-84f5-ce5550fd76ae" (UID: "b6b9fc40-2379-498d-84f5-ce5550fd76ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:44.295868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.295575 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-workload-certs" pod "b6b9fc40-2379-498d-84f5-ce5550fd76ae" (UID: "b6b9fc40-2379-498d-84f5-ce5550fd76ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:44.295868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.295632 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istiod-ca-cert" pod "b6b9fc40-2379-498d-84f5-ce5550fd76ae" (UID: "b6b9fc40-2379-498d-84f5-ce5550fd76ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:44.295868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.295746 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:44.295868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.295763 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:44.295868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.295773 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:44.295868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.295786 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:44.295868 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.295798 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:44.298313 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.298281 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b9fc40-2379-498d-84f5-ce5550fd76ae-kube-api-access-zdfpx" pod "b6b9fc40-2379-498d-84f5-ce5550fd76ae" (UID: "b6b9fc40-2379-498d-84f5-ce5550fd76ae"). InnerVolumeSpecName "kube-api-access-zdfpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:44.298400 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.298306 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-token" pod "b6b9fc40-2379-498d-84f5-ce5550fd76ae" (UID: "b6b9fc40-2379-498d-84f5-ce5550fd76ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:44.298400 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.298378 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-podinfo" pod "b6b9fc40-2379-498d-84f5-ce5550fd76ae" (UID: "b6b9fc40-2379-498d-84f5-ce5550fd76ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 03:59:44.298861 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.298840 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-envoy" pod "b6b9fc40-2379-498d-84f5-ce5550fd76ae" (UID: "b6b9fc40-2379-498d-84f5-ce5550fd76ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 03:59:44.312460 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.312425 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:44.396966 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.396933 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:44.396966 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.396961 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:44.396966 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.396974 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdfpx\" (UniqueName: \"kubernetes.io/projected/b6b9fc40-2379-498d-84f5-ce5550fd76ae-kube-api-access-zdfpx\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:44.397224 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.396983 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b9fc40-2379-498d-84f5-ce5550fd76ae-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:44.752855 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.752816 5490 generic.go:356] "Generic (PLEG): container finished" podID="b6b9fc40-2379-498d-84f5-ce5550fd76ae" containerID="73c2237fb7b9b625272b3c052a08a9cadcaefe16179dc32b50a020dc070f2855" exitCode=0 Oct 31 03:59:44.753078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.752871 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" Oct 31 03:59:44.753078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.752902 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" event={"ID":"b6b9fc40-2379-498d-84f5-ce5550fd76ae","Type":"ContainerDied","Data":"73c2237fb7b9b625272b3c052a08a9cadcaefe16179dc32b50a020dc070f2855"} Oct 31 03:59:44.753078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.752973 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj" event={"ID":"b6b9fc40-2379-498d-84f5-ce5550fd76ae","Type":"ContainerDied","Data":"1a80b56d6f0f5a85f4dbb55fe5cc6791df755429752a38fddb3a1fd226a98254"} Oct 31 03:59:44.753078 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.753004 5490 scope.go:122] "RemoveContainer" containerID="73c2237fb7b9b625272b3c052a08a9cadcaefe16179dc32b50a020dc070f2855" Oct 31 03:59:44.764305 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.764280 5490 scope.go:122] "RemoveContainer" containerID="73c2237fb7b9b625272b3c052a08a9cadcaefe16179dc32b50a020dc070f2855" Oct 31 03:59:44.764624 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:59:44.764599 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73c2237fb7b9b625272b3c052a08a9cadcaefe16179dc32b50a020dc070f2855\": container with ID starting with 73c2237fb7b9b625272b3c052a08a9cadcaefe16179dc32b50a020dc070f2855 not found: ID does not exist" containerID="73c2237fb7b9b625272b3c052a08a9cadcaefe16179dc32b50a020dc070f2855" Oct 31 03:59:44.764734 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.764632 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c2237fb7b9b625272b3c052a08a9cadcaefe16179dc32b50a020dc070f2855"} err="failed to get container status \"73c2237fb7b9b625272b3c052a08a9cadcaefe16179dc32b50a020dc070f2855\": rpc error: code = NotFound desc = could not find container \"73c2237fb7b9b625272b3c052a08a9cadcaefe16179dc32b50a020dc070f2855\": container with ID starting with 73c2237fb7b9b625272b3c052a08a9cadcaefe16179dc32b50a020dc070f2855 not found: ID does not exist" Oct 31 03:59:44.776419 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.776387 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj"] Oct 31 03:59:44.785959 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.785916 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-2g6tj"] Oct 31 03:59:44.986644 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:44.986605 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:45.575597 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:45.575561 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b9fc40-2379-498d-84f5-ce5550fd76ae" path="/var/lib/kubelet/pods/b6b9fc40-2379-498d-84f5-ce5550fd76ae/volumes" Oct 31 03:59:45.593154 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:45.593113 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:46.184898 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:46.184856 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:46.802168 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:46.802121 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:47.128760 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:59:47.128665 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 03:59:47.409133 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:47.409095 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:47.996506 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:47.996306 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:48.657669 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:48.657628 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:48.790716 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:48.790639 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:49.159016 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:49.158974 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" podUID="42e53881-926d-482d-9557-a86d3ae3f9d9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 03:59:49.296203 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:49.296166 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:49.707336 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:49.707300 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:49.840756 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:49.840713 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:50.017994 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:50.017915 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:50.783326 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:50.783130 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:51.411268 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.411227 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74"] Oct 31 03:59:51.411586 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.411523 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" podUID="e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff" containerName="maas-api" containerID="cri-o://e3b25ee8fc97e7bbd6f93e95feb4abe217e500bc811341e1056c60ab3e0675cd" gracePeriod=30 Oct 31 03:59:51.496921 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.496885 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:51.590578 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.590445 5490 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod24a03b21-4c4f-422e-baa4-6e905b1b0bf5"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod24a03b21-4c4f-422e-baa4-6e905b1b0bf5] : Timed out while waiting for systemd to remove kubepods-burstable-pod24a03b21_4c4f_422e_baa4_6e905b1b0bf5.slice" Oct 31 03:59:51.590578 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.590527 5490 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="24a03b21-4c4f-422e-baa4-6e905b1b0bf5" err="failed to delete cgroup paths for [kubepods burstable pod24a03b21-4c4f-422e-baa4-6e905b1b0bf5] : unable to destroy cgroup paths for cgroup [kubepods burstable pod24a03b21-4c4f-422e-baa4-6e905b1b0bf5] : Timed out while waiting for systemd to remove kubepods-burstable-pod24a03b21_4c4f_422e_baa4_6e905b1b0bf5.slice" Oct 31 03:59:51.648216 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.648175 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:51.677461 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.677395 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" Oct 31 03:59:51.776004 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.775963 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-maas-api-tls\") pod \"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff\" (UID: \"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff\") " Oct 31 03:59:51.776232 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.776075 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-openshift-service-ca\") pod \"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff\" (UID: \"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff\") " Oct 31 03:59:51.776232 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.776182 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-kube-api-access-7x4mm\" (UniqueName: \"kubernetes.io/projected/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-kube-api-access-7x4mm\") pod \"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff\" (UID: \"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff\") " Oct 31 03:59:51.776362 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.776261 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-metrics-certs\") pod \"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff\" (UID: \"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff\") " Oct 31 03:59:51.776586 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.776558 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-openshift-service-ca" pod "e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff" (UID: "e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:51.776804 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.776772 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:51.778459 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.778431 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-metrics-certs" pod "e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff" (UID: "e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:51.778459 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.778442 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-kube-api-access-7x4mm" pod "e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff" (UID: "e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff"). InnerVolumeSpecName "kube-api-access-7x4mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:51.778629 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.778529 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-maas-api-tls" pod "e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff" (UID: "e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 03:59:51.784772 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.784658 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:51.796874 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.796837 5490 generic.go:356] "Generic (PLEG): container finished" podID="e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff" containerID="e3b25ee8fc97e7bbd6f93e95feb4abe217e500bc811341e1056c60ab3e0675cd" exitCode=0 Oct 31 03:59:51.797016 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.796886 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" Oct 31 03:59:51.797016 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.796893 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" event={"ID":"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff","Type":"ContainerDied","Data":"e3b25ee8fc97e7bbd6f93e95feb4abe217e500bc811341e1056c60ab3e0675cd"} Oct 31 03:59:51.797016 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.796953 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74" event={"ID":"e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff","Type":"ContainerDied","Data":"776dea0842237cc39ee362f893edbac0a082fca3837d3037c6aefedcf4525f3c"} Oct 31 03:59:51.797016 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.796983 5490 scope.go:122] "RemoveContainer" containerID="e3b25ee8fc97e7bbd6f93e95feb4abe217e500bc811341e1056c60ab3e0675cd" Oct 31 03:59:51.808175 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.808150 5490 scope.go:122] "RemoveContainer" containerID="e3b25ee8fc97e7bbd6f93e95feb4abe217e500bc811341e1056c60ab3e0675cd" Oct 31 03:59:51.808474 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:59:51.808454 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3b25ee8fc97e7bbd6f93e95feb4abe217e500bc811341e1056c60ab3e0675cd\": container with ID starting with e3b25ee8fc97e7bbd6f93e95feb4abe217e500bc811341e1056c60ab3e0675cd not found: ID does not exist" containerID="e3b25ee8fc97e7bbd6f93e95feb4abe217e500bc811341e1056c60ab3e0675cd" Oct 31 03:59:51.808529 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.808486 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3b25ee8fc97e7bbd6f93e95feb4abe217e500bc811341e1056c60ab3e0675cd"} err="failed to get container status \"e3b25ee8fc97e7bbd6f93e95feb4abe217e500bc811341e1056c60ab3e0675cd\": rpc error: code = NotFound desc = could not find container \"e3b25ee8fc97e7bbd6f93e95feb4abe217e500bc811341e1056c60ab3e0675cd\": container with ID starting with e3b25ee8fc97e7bbd6f93e95feb4abe217e500bc811341e1056c60ab3e0675cd not found: ID does not exist" Oct 31 03:59:51.820152 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.820124 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74"] Oct 31 03:59:51.824141 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.824103 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-ab21e4-77df9c566c-b4q74"] Oct 31 03:59:51.877732 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.877697 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x4mm\" (UniqueName: \"kubernetes.io/projected/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-kube-api-access-7x4mm\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:51.877732 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.877730 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:51.877961 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:51.877743 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:52.254449 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:52.254407 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:52.599955 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:52.599869 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:53.278689 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:53.278640 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:53.351502 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:53.351441 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" Oct 31 03:59:53.391187 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:53.391150 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42e53881-926d-482d-9557-a86d3ae3f9d9-kube-api-access-zn78k\" (UniqueName: \"kubernetes.io/projected/42e53881-926d-482d-9557-a86d3ae3f9d9-kube-api-access-zn78k\") pod \"42e53881-926d-482d-9557-a86d3ae3f9d9\" (UID: \"42e53881-926d-482d-9557-a86d3ae3f9d9\") " Oct 31 03:59:53.391387 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:53.391224 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42e53881-926d-482d-9557-a86d3ae3f9d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e53881-926d-482d-9557-a86d3ae3f9d9-plugins-config-volume\") pod \"42e53881-926d-482d-9557-a86d3ae3f9d9\" (UID: \"42e53881-926d-482d-9557-a86d3ae3f9d9\") " Oct 31 03:59:53.391740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:53.391709 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42e53881-926d-482d-9557-a86d3ae3f9d9-plugins-config-volume" pod "42e53881-926d-482d-9557-a86d3ae3f9d9" (UID: "42e53881-926d-482d-9557-a86d3ae3f9d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 03:59:53.393462 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:53.393431 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e53881-926d-482d-9557-a86d3ae3f9d9-kube-api-access-zn78k" pod "42e53881-926d-482d-9557-a86d3ae3f9d9" (UID: "42e53881-926d-482d-9557-a86d3ae3f9d9"). InnerVolumeSpecName "kube-api-access-zn78k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 03:59:53.492669 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:53.492627 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn78k\" (UniqueName: \"kubernetes.io/projected/42e53881-926d-482d-9557-a86d3ae3f9d9-kube-api-access-zn78k\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:53.492669 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:53.492665 5490 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42e53881-926d-482d-9557-a86d3ae3f9d9-plugins-config-volume\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 03:59:53.574168 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:53.574137 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff" path="/var/lib/kubelet/pods/e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff/volumes" Oct 31 03:59:53.811636 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:53.811599 5490 generic.go:356] "Generic (PLEG): container finished" podID="42e53881-926d-482d-9557-a86d3ae3f9d9" containerID="dfe524b27adecb16940953181c7b9e812fd05a362305d00c551e3c6d3d67a10b" exitCode=0 Oct 31 03:59:53.811892 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:53.811653 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" Oct 31 03:59:53.811892 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:53.811707 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" event={"ID":"42e53881-926d-482d-9557-a86d3ae3f9d9","Type":"ContainerDied","Data":"dfe524b27adecb16940953181c7b9e812fd05a362305d00c551e3c6d3d67a10b"} Oct 31 03:59:53.811892 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:53.811770 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw" event={"ID":"42e53881-926d-482d-9557-a86d3ae3f9d9","Type":"ContainerDied","Data":"f0cda0ed41511dbe873234782faa3da6d83f41e4029bfd84e48a591cd3f6c2b6"} Oct 31 03:59:53.811892 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:53.811795 5490 scope.go:122] "RemoveContainer" containerID="dfe524b27adecb16940953181c7b9e812fd05a362305d00c551e3c6d3d67a10b" Oct 31 03:59:53.828937 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:53.828907 5490 scope.go:122] "RemoveContainer" containerID="dfe524b27adecb16940953181c7b9e812fd05a362305d00c551e3c6d3d67a10b" Oct 31 03:59:53.829382 ip-10-0-142-76 kubenswrapper[5490]: E1031 03:59:53.829344 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe524b27adecb16940953181c7b9e812fd05a362305d00c551e3c6d3d67a10b\": container with ID starting with dfe524b27adecb16940953181c7b9e812fd05a362305d00c551e3c6d3d67a10b not found: ID does not exist" containerID="dfe524b27adecb16940953181c7b9e812fd05a362305d00c551e3c6d3d67a10b" Oct 31 03:59:53.829525 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:53.829392 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe524b27adecb16940953181c7b9e812fd05a362305d00c551e3c6d3d67a10b"} err="failed to get container status \"dfe524b27adecb16940953181c7b9e812fd05a362305d00c551e3c6d3d67a10b\": rpc error: code = NotFound desc = could not find container \"dfe524b27adecb16940953181c7b9e812fd05a362305d00c551e3c6d3d67a10b\": container with ID starting with dfe524b27adecb16940953181c7b9e812fd05a362305d00c551e3c6d3d67a10b not found: ID does not exist" Oct 31 03:59:53.830468 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:53.830445 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw"] Oct 31 03:59:53.832887 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:53.832868 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-34cb76-5cdff4c64c-p4vqw"] Oct 31 03:59:54.075391 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:54.075288 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:54.694520 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:54.692640 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:55.431352 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:55.431312 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:55.574325 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:55.574286 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42e53881-926d-482d-9557-a86d3ae3f9d9" path="/var/lib/kubelet/pods/42e53881-926d-482d-9557-a86d3ae3f9d9/volumes" Oct 31 03:59:56.262541 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:56.262506 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:57.076040 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:57.076000 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:57.702074 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:57.702026 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:58.309740 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:58.309671 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:58.643362 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:58.643270 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:58.795219 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:58.795176 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:58.921041 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:58.921005 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq"] Oct 31 03:59:58.921267 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:58.921220 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" podUID="a8a2a69d-e4b7-4338-a7b4-22042b94432f" containerName="istio-proxy" containerID="cri-o://4d01c9e993f57b94598c5d6e7b7c4abb8bc04cae0370ec6131a24bcfede02400" gracePeriod=30 Oct 31 03:59:59.019088 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:59.019040 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:59.156495 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:59.156448 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:59.504388 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:59.504350 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 03:59:59.863729 ip-10-0-142-76 kubenswrapper[5490]: I1031 03:59:59.863598 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:00.679383 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:00.679344 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n"] Oct 31 04:00:00.680008 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:00.679562 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" podUID="1303047d-c782-41c7-bc35-d3298c9ef224" containerName="maas-api" containerID="cri-o://3d916d97247ba656eebfa4e5db031bdc9b1855ddaeb6bd8d2946f85ddf648907" gracePeriod=30 Oct 31 04:00:00.689603 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:00.689564 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" podUID="1303047d-c782-41c7-bc35-d3298c9ef224" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.62:8443/health\": dial tcp 10.133.0.62:8443: connect: connection refused" Oct 31 04:00:00.777972 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:00.777937 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:00.865261 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:00.865224 5490 generic.go:356] "Generic (PLEG): container finished" podID="1303047d-c782-41c7-bc35-d3298c9ef224" containerID="3d916d97247ba656eebfa4e5db031bdc9b1855ddaeb6bd8d2946f85ddf648907" exitCode=0 Oct 31 04:00:00.865622 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:00.865301 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" event={"ID":"1303047d-c782-41c7-bc35-d3298c9ef224","Type":"ContainerDied","Data":"3d916d97247ba656eebfa4e5db031bdc9b1855ddaeb6bd8d2946f85ddf648907"} Oct 31 04:00:00.946355 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:00.946274 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" Oct 31 04:00:01.011924 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.011888 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:01.078907 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.078873 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1303047d-c782-41c7-bc35-d3298c9ef224-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1303047d-c782-41c7-bc35-d3298c9ef224-maas-api-tls\") pod \"1303047d-c782-41c7-bc35-d3298c9ef224\" (UID: \"1303047d-c782-41c7-bc35-d3298c9ef224\") " Oct 31 04:00:01.079102 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.078947 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1303047d-c782-41c7-bc35-d3298c9ef224-kube-api-access-hmwxm\" (UniqueName: \"kubernetes.io/projected/1303047d-c782-41c7-bc35-d3298c9ef224-kube-api-access-hmwxm\") pod \"1303047d-c782-41c7-bc35-d3298c9ef224\" (UID: \"1303047d-c782-41c7-bc35-d3298c9ef224\") " Oct 31 04:00:01.079102 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.079011 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1303047d-c782-41c7-bc35-d3298c9ef224-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1303047d-c782-41c7-bc35-d3298c9ef224-metrics-certs\") pod \"1303047d-c782-41c7-bc35-d3298c9ef224\" (UID: \"1303047d-c782-41c7-bc35-d3298c9ef224\") " Oct 31 04:00:01.079102 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.079047 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1303047d-c782-41c7-bc35-d3298c9ef224-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1303047d-c782-41c7-bc35-d3298c9ef224-openshift-service-ca\") pod \"1303047d-c782-41c7-bc35-d3298c9ef224\" (UID: \"1303047d-c782-41c7-bc35-d3298c9ef224\") " Oct 31 04:00:01.079515 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.079484 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1303047d-c782-41c7-bc35-d3298c9ef224-openshift-service-ca" pod "1303047d-c782-41c7-bc35-d3298c9ef224" (UID: "1303047d-c782-41c7-bc35-d3298c9ef224"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:01.081193 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.081160 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1303047d-c782-41c7-bc35-d3298c9ef224-maas-api-tls" pod "1303047d-c782-41c7-bc35-d3298c9ef224" (UID: "1303047d-c782-41c7-bc35-d3298c9ef224"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:00:01.081297 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.081186 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1303047d-c782-41c7-bc35-d3298c9ef224-metrics-certs" pod "1303047d-c782-41c7-bc35-d3298c9ef224" (UID: "1303047d-c782-41c7-bc35-d3298c9ef224"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:00:01.081297 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.081190 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1303047d-c782-41c7-bc35-d3298c9ef224-kube-api-access-hmwxm" pod "1303047d-c782-41c7-bc35-d3298c9ef224" (UID: "1303047d-c782-41c7-bc35-d3298c9ef224"). InnerVolumeSpecName "kube-api-access-hmwxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:01.180629 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.180597 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1303047d-c782-41c7-bc35-d3298c9ef224-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:01.180629 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.180630 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmwxm\" (UniqueName: \"kubernetes.io/projected/1303047d-c782-41c7-bc35-d3298c9ef224-kube-api-access-hmwxm\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:01.180866 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.180646 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1303047d-c782-41c7-bc35-d3298c9ef224-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:01.180866 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.180660 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1303047d-c782-41c7-bc35-d3298c9ef224-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:01.195776 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.195742 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:01.329135 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.328948 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:01.539772 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.539736 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:01.872170 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.872090 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" Oct 31 04:00:01.872170 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.872127 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n" event={"ID":"1303047d-c782-41c7-bc35-d3298c9ef224","Type":"ContainerDied","Data":"dbf657a00c699bcc04cd3b856f64e09f374b097469824cd5d6a27d6662ed381d"} Oct 31 04:00:01.872717 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.872189 5490 scope.go:122] "RemoveContainer" containerID="3d916d97247ba656eebfa4e5db031bdc9b1855ddaeb6bd8d2946f85ddf648907" Oct 31 04:00:01.893468 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.893423 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n"] Oct 31 04:00:01.895960 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:01.895934 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-7f3159-9dd965484-z7w8n"] Oct 31 04:00:02.096063 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:02.096023 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:02.698538 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:02.698494 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:03.498288 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:03.498249 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:03.573811 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:03.573781 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1303047d-c782-41c7-bc35-d3298c9ef224" path="/var/lib/kubelet/pods/1303047d-c782-41c7-bc35-d3298c9ef224/volumes" Oct 31 04:00:04.182583 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.182558 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 04:00:04.203838 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.203762 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:04.313940 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.313904 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-data\") pod \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " Oct 31 04:00:04.314140 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.313954 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-podinfo\") pod \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " Oct 31 04:00:04.314140 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.313983 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istiod-ca-cert\") pod \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " Oct 31 04:00:04.314140 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.314015 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-credential-socket\") pod \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " Oct 31 04:00:04.314140 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.314040 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-envoy\") pod \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " Oct 31 04:00:04.314140 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.314078 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a2a69d-e4b7-4338-a7b4-22042b94432f-kube-api-access-vk7f5\" (UniqueName: \"kubernetes.io/projected/a8a2a69d-e4b7-4338-a7b4-22042b94432f-kube-api-access-vk7f5\") pod \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " Oct 31 04:00:04.314409 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.314175 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-token\" (UniqueName: \"kubernetes.io/projected/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-token\") pod \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " Oct 31 04:00:04.314409 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.314257 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-workload-socket\") pod \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " Oct 31 04:00:04.314409 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.314272 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-data" pod "a8a2a69d-e4b7-4338-a7b4-22042b94432f" (UID: "a8a2a69d-e4b7-4338-a7b4-22042b94432f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:04.314409 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.314303 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-workload-certs\") pod \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\" (UID: \"a8a2a69d-e4b7-4338-a7b4-22042b94432f\") " Oct 31 04:00:04.314625 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.314509 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-workload-socket" pod "a8a2a69d-e4b7-4338-a7b4-22042b94432f" (UID: "a8a2a69d-e4b7-4338-a7b4-22042b94432f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:04.314625 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.314534 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-credential-socket" pod "a8a2a69d-e4b7-4338-a7b4-22042b94432f" (UID: "a8a2a69d-e4b7-4338-a7b4-22042b94432f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:04.314625 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.314575 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istiod-ca-cert" pod "a8a2a69d-e4b7-4338-a7b4-22042b94432f" (UID: "a8a2a69d-e4b7-4338-a7b4-22042b94432f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:04.314839 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.314734 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-workload-certs" pod "a8a2a69d-e4b7-4338-a7b4-22042b94432f" (UID: "a8a2a69d-e4b7-4338-a7b4-22042b94432f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:04.314839 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.314827 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:04.314930 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.314850 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:04.314930 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.314865 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:04.314930 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.314880 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:04.316868 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.316834 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-envoy" pod "a8a2a69d-e4b7-4338-a7b4-22042b94432f" (UID: "a8a2a69d-e4b7-4338-a7b4-22042b94432f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:04.317012 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.316993 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-podinfo" pod "a8a2a69d-e4b7-4338-a7b4-22042b94432f" (UID: "a8a2a69d-e4b7-4338-a7b4-22042b94432f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:00:04.317012 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.317006 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-token" pod "a8a2a69d-e4b7-4338-a7b4-22042b94432f" (UID: "a8a2a69d-e4b7-4338-a7b4-22042b94432f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:04.317096 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.316994 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a2a69d-e4b7-4338-a7b4-22042b94432f-kube-api-access-vk7f5" pod "a8a2a69d-e4b7-4338-a7b4-22042b94432f" (UID: "a8a2a69d-e4b7-4338-a7b4-22042b94432f"). InnerVolumeSpecName "kube-api-access-vk7f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:04.415524 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.415484 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:04.415524 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.415522 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:04.415524 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.415532 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:04.415827 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.415541 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk7f5\" (UniqueName: \"kubernetes.io/projected/a8a2a69d-e4b7-4338-a7b4-22042b94432f-kube-api-access-vk7f5\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:04.415827 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.415551 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8a2a69d-e4b7-4338-a7b4-22042b94432f-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:04.423563 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.423528 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl"] Oct 31 04:00:04.536640 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.536555 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:04.894985 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.894882 5490 generic.go:356] "Generic (PLEG): container finished" podID="a8a2a69d-e4b7-4338-a7b4-22042b94432f" containerID="4d01c9e993f57b94598c5d6e7b7c4abb8bc04cae0370ec6131a24bcfede02400" exitCode=0 Oct 31 04:00:04.894985 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.894950 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" Oct 31 04:00:04.894985 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.894967 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" event={"ID":"a8a2a69d-e4b7-4338-a7b4-22042b94432f","Type":"ContainerDied","Data":"4d01c9e993f57b94598c5d6e7b7c4abb8bc04cae0370ec6131a24bcfede02400"} Oct 31 04:00:04.895244 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.895027 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq" event={"ID":"a8a2a69d-e4b7-4338-a7b4-22042b94432f","Type":"ContainerDied","Data":"0562dfe171ae30048b707d66b745f588cd3729f4a6d22e7c45fad7d57f276e71"} Oct 31 04:00:04.895244 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.895052 5490 scope.go:122] "RemoveContainer" containerID="4d01c9e993f57b94598c5d6e7b7c4abb8bc04cae0370ec6131a24bcfede02400" Oct 31 04:00:04.906330 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.906306 5490 scope.go:122] "RemoveContainer" containerID="b1fd0c247d873205c8e76e9c6e1b1c75b4d4095c36cc0b09c203f58771e7431c" Oct 31 04:00:04.917218 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.917192 5490 scope.go:122] "RemoveContainer" containerID="4d01c9e993f57b94598c5d6e7b7c4abb8bc04cae0370ec6131a24bcfede02400" Oct 31 04:00:04.917589 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:00:04.917565 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d01c9e993f57b94598c5d6e7b7c4abb8bc04cae0370ec6131a24bcfede02400\": container with ID starting with 4d01c9e993f57b94598c5d6e7b7c4abb8bc04cae0370ec6131a24bcfede02400 not found: ID does not exist" containerID="4d01c9e993f57b94598c5d6e7b7c4abb8bc04cae0370ec6131a24bcfede02400" Oct 31 04:00:04.917768 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.917596 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d01c9e993f57b94598c5d6e7b7c4abb8bc04cae0370ec6131a24bcfede02400"} err="failed to get container status \"4d01c9e993f57b94598c5d6e7b7c4abb8bc04cae0370ec6131a24bcfede02400\": rpc error: code = NotFound desc = could not find container \"4d01c9e993f57b94598c5d6e7b7c4abb8bc04cae0370ec6131a24bcfede02400\": container with ID starting with 4d01c9e993f57b94598c5d6e7b7c4abb8bc04cae0370ec6131a24bcfede02400 not found: ID does not exist" Oct 31 04:00:04.917768 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.917614 5490 scope.go:122] "RemoveContainer" containerID="b1fd0c247d873205c8e76e9c6e1b1c75b4d4095c36cc0b09c203f58771e7431c" Oct 31 04:00:04.917952 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:00:04.917933 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1fd0c247d873205c8e76e9c6e1b1c75b4d4095c36cc0b09c203f58771e7431c\": container with ID starting with b1fd0c247d873205c8e76e9c6e1b1c75b4d4095c36cc0b09c203f58771e7431c not found: ID does not exist" containerID="b1fd0c247d873205c8e76e9c6e1b1c75b4d4095c36cc0b09c203f58771e7431c" Oct 31 04:00:04.918018 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.917955 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1fd0c247d873205c8e76e9c6e1b1c75b4d4095c36cc0b09c203f58771e7431c"} err="failed to get container status \"b1fd0c247d873205c8e76e9c6e1b1c75b4d4095c36cc0b09c203f58771e7431c\": rpc error: code = NotFound desc = could not find container \"b1fd0c247d873205c8e76e9c6e1b1c75b4d4095c36cc0b09c203f58771e7431c\": container with ID starting with b1fd0c247d873205c8e76e9c6e1b1c75b4d4095c36cc0b09c203f58771e7431c not found: ID does not exist" Oct 31 04:00:04.919405 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.919375 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq"] Oct 31 04:00:04.925434 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.925394 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-ab21e4-openshift-default-698bdbf5cc-4jcgq"] Oct 31 04:00:04.980650 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:04.980610 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:05.576105 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:05.576074 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8a2a69d-e4b7-4338-a7b4-22042b94432f" path="/var/lib/kubelet/pods/a8a2a69d-e4b7-4338-a7b4-22042b94432f/volumes" Oct 31 04:00:05.654046 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:05.654009 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:06.192623 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:06.192518 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:06.892515 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:06.892469 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:07.703475 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:07.703438 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:08.406313 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:08.406266 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:08.547850 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:08.547811 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:08.676147 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:08.676057 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:08.932283 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:08.932200 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:09.288668 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:09.288570 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:09.924803 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:09.924770 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:10.055424 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:10.055386 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:10.181090 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:10.181003 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:17.128567 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:00:17.128524 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:00:18.542191 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:18.542153 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:19.444819 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.444773 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" containerID="cri-o://cd1a44801f0be21c518fb9f1e10bc9448b6c2057a17cc96369e9b39ffece50af" gracePeriod=45 Oct 31 04:00:19.485940 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.485901 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": dial tcp 10.133.0.64:8000: connect: connection refused" Oct 31 04:00:19.703632 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.703564 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 04:00:19.767829 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.767780 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-dshm\") pod \"1712a46e-a034-40f6-a276-09a907dd082f\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " Oct 31 04:00:19.768047 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.767855 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1712a46e-a034-40f6-a276-09a907dd082f-tls-certs\" (UniqueName: \"kubernetes.io/secret/1712a46e-a034-40f6-a276-09a907dd082f-tls-certs\") pod \"1712a46e-a034-40f6-a276-09a907dd082f\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " Oct 31 04:00:19.768047 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.767949 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-model-cache\") pod \"1712a46e-a034-40f6-a276-09a907dd082f\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " Oct 31 04:00:19.768211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.768078 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-tmp-dir\") pod \"1712a46e-a034-40f6-a276-09a907dd082f\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " Oct 31 04:00:19.768211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.768123 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1712a46e-a034-40f6-a276-09a907dd082f-kube-api-access-n246z\" (UniqueName: \"kubernetes.io/projected/1712a46e-a034-40f6-a276-09a907dd082f-kube-api-access-n246z\") pod \"1712a46e-a034-40f6-a276-09a907dd082f\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " Oct 31 04:00:19.768211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.768174 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-home\" (UniqueName: \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-home\") pod \"1712a46e-a034-40f6-a276-09a907dd082f\" (UID: \"1712a46e-a034-40f6-a276-09a907dd082f\") " Oct 31 04:00:19.768393 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.768233 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-model-cache" pod "1712a46e-a034-40f6-a276-09a907dd082f" (UID: "1712a46e-a034-40f6-a276-09a907dd082f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:19.768481 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.768454 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-tmp-dir" pod "1712a46e-a034-40f6-a276-09a907dd082f" (UID: "1712a46e-a034-40f6-a276-09a907dd082f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:19.768779 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.768749 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-home" pod "1712a46e-a034-40f6-a276-09a907dd082f" (UID: "1712a46e-a034-40f6-a276-09a907dd082f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:19.768960 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.768791 5490 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-model-cache\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:19.768960 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.768844 5490 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-tmp-dir\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:19.770642 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.770613 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1712a46e-a034-40f6-a276-09a907dd082f-tls-certs" pod "1712a46e-a034-40f6-a276-09a907dd082f" (UID: "1712a46e-a034-40f6-a276-09a907dd082f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:00:19.770898 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.770875 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-dshm" pod "1712a46e-a034-40f6-a276-09a907dd082f" (UID: "1712a46e-a034-40f6-a276-09a907dd082f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:00:19.771153 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.771128 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1712a46e-a034-40f6-a276-09a907dd082f-kube-api-access-n246z" pod "1712a46e-a034-40f6-a276-09a907dd082f" (UID: "1712a46e-a034-40f6-a276-09a907dd082f"). InnerVolumeSpecName "kube-api-access-n246z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:19.870285 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.870243 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n246z\" (UniqueName: \"kubernetes.io/projected/1712a46e-a034-40f6-a276-09a907dd082f-kube-api-access-n246z\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:19.870285 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.870280 5490 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-home\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:19.870285 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.870295 5490 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1712a46e-a034-40f6-a276-09a907dd082f-dshm\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:19.870550 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.870306 5490 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1712a46e-a034-40f6-a276-09a907dd082f-tls-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:19.986802 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.986715 5490 generic.go:356] "Generic (PLEG): container finished" podID="1712a46e-a034-40f6-a276-09a907dd082f" containerID="cd1a44801f0be21c518fb9f1e10bc9448b6c2057a17cc96369e9b39ffece50af" exitCode=0 Oct 31 04:00:19.986802 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.986769 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" Oct 31 04:00:19.987017 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.986770 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" event={"ID":"1712a46e-a034-40f6-a276-09a907dd082f","Type":"ContainerDied","Data":"cd1a44801f0be21c518fb9f1e10bc9448b6c2057a17cc96369e9b39ffece50af"} Oct 31 04:00:19.987017 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.986882 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl" event={"ID":"1712a46e-a034-40f6-a276-09a907dd082f","Type":"ContainerDied","Data":"e3c59306b82ea5223a1848b314404162d3337758f1f1eb395202afa94d2f337a"} Oct 31 04:00:19.987017 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.986899 5490 scope.go:122] "RemoveContainer" containerID="cd1a44801f0be21c518fb9f1e10bc9448b6c2057a17cc96369e9b39ffece50af" Oct 31 04:00:19.997332 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.997312 5490 scope.go:122] "RemoveContainer" containerID="cd1a44801f0be21c518fb9f1e10bc9448b6c2057a17cc96369e9b39ffece50af" Oct 31 04:00:19.997641 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:00:19.997618 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd1a44801f0be21c518fb9f1e10bc9448b6c2057a17cc96369e9b39ffece50af\": container with ID starting with cd1a44801f0be21c518fb9f1e10bc9448b6c2057a17cc96369e9b39ffece50af not found: ID does not exist" containerID="cd1a44801f0be21c518fb9f1e10bc9448b6c2057a17cc96369e9b39ffece50af" Oct 31 04:00:19.997747 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:19.997650 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd1a44801f0be21c518fb9f1e10bc9448b6c2057a17cc96369e9b39ffece50af"} err="failed to get container status \"cd1a44801f0be21c518fb9f1e10bc9448b6c2057a17cc96369e9b39ffece50af\": rpc error: code = NotFound desc = could not find container \"cd1a44801f0be21c518fb9f1e10bc9448b6c2057a17cc96369e9b39ffece50af\": container with ID starting with cd1a44801f0be21c518fb9f1e10bc9448b6c2057a17cc96369e9b39ffece50af not found: ID does not exist" Oct 31 04:00:20.012112 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:20.012081 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl"] Oct 31 04:00:20.017030 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:20.017000 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/premium-w5-bcc7a4-kserve-864876c669-wlghl"] Oct 31 04:00:20.896472 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:20.896426 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:21.049104 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:21.049065 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:21.201545 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:21.201487 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:21.337044 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:21.337003 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:21.574149 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:21.574070 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1712a46e-a034-40f6-a276-09a907dd082f" path="/var/lib/kubelet/pods/1712a46e-a034-40f6-a276-09a907dd082f/volumes" Oct 31 04:00:21.831220 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:21.831119 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:25.616969 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:25.616931 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5"] Oct 31 04:00:25.617467 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:25.617180 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" podUID="531c8516-8a62-4add-a9bb-15b02ebb6f5d" containerName="maas-api" containerID="cri-o://db46fc3bf6d0814a7e7d227a3141110ba8ea1f0ac6d031c66ca2a2318bfa2ee1" gracePeriod=30 Oct 31 04:00:25.690172 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:25.690122 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" podUID="531c8516-8a62-4add-a9bb-15b02ebb6f5d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.61:8443/health\": dial tcp 10.133.0.61:8443: connect: connection refused" Oct 31 04:00:25.699010 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:25.698970 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:25.859177 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:25.859139 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:25.879275 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:25.879212 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" Oct 31 04:00:25.929051 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:25.929015 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/531c8516-8a62-4add-a9bb-15b02ebb6f5d-kube-api-access-7zfzn\" (UniqueName: \"kubernetes.io/projected/531c8516-8a62-4add-a9bb-15b02ebb6f5d-kube-api-access-7zfzn\") pod \"531c8516-8a62-4add-a9bb-15b02ebb6f5d\" (UID: \"531c8516-8a62-4add-a9bb-15b02ebb6f5d\") " Oct 31 04:00:25.929257 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:25.929122 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/531c8516-8a62-4add-a9bb-15b02ebb6f5d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/531c8516-8a62-4add-a9bb-15b02ebb6f5d-metrics-certs\") pod \"531c8516-8a62-4add-a9bb-15b02ebb6f5d\" (UID: \"531c8516-8a62-4add-a9bb-15b02ebb6f5d\") " Oct 31 04:00:25.929257 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:25.929210 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/531c8516-8a62-4add-a9bb-15b02ebb6f5d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531c8516-8a62-4add-a9bb-15b02ebb6f5d-openshift-service-ca\") pod \"531c8516-8a62-4add-a9bb-15b02ebb6f5d\" (UID: \"531c8516-8a62-4add-a9bb-15b02ebb6f5d\") " Oct 31 04:00:25.929257 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:25.929236 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/531c8516-8a62-4add-a9bb-15b02ebb6f5d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531c8516-8a62-4add-a9bb-15b02ebb6f5d-maas-api-tls\") pod \"531c8516-8a62-4add-a9bb-15b02ebb6f5d\" (UID: \"531c8516-8a62-4add-a9bb-15b02ebb6f5d\") " Oct 31 04:00:25.929809 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:25.929773 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/531c8516-8a62-4add-a9bb-15b02ebb6f5d-openshift-service-ca" pod "531c8516-8a62-4add-a9bb-15b02ebb6f5d" (UID: "531c8516-8a62-4add-a9bb-15b02ebb6f5d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:00:25.931465 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:25.931382 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531c8516-8a62-4add-a9bb-15b02ebb6f5d-kube-api-access-7zfzn" pod "531c8516-8a62-4add-a9bb-15b02ebb6f5d" (UID: "531c8516-8a62-4add-a9bb-15b02ebb6f5d"). InnerVolumeSpecName "kube-api-access-7zfzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:25.931582 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:25.931556 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/531c8516-8a62-4add-a9bb-15b02ebb6f5d-metrics-certs" pod "531c8516-8a62-4add-a9bb-15b02ebb6f5d" (UID: "531c8516-8a62-4add-a9bb-15b02ebb6f5d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:00:25.931864 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:25.931834 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/531c8516-8a62-4add-a9bb-15b02ebb6f5d-maas-api-tls" pod "531c8516-8a62-4add-a9bb-15b02ebb6f5d" (UID: "531c8516-8a62-4add-a9bb-15b02ebb6f5d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:00:25.986124 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:25.986088 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:26.023030 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:26.022998 5490 generic.go:356] "Generic (PLEG): container finished" podID="531c8516-8a62-4add-a9bb-15b02ebb6f5d" containerID="db46fc3bf6d0814a7e7d227a3141110ba8ea1f0ac6d031c66ca2a2318bfa2ee1" exitCode=0 Oct 31 04:00:26.023237 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:26.023051 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" Oct 31 04:00:26.023237 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:26.023078 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" event={"ID":"531c8516-8a62-4add-a9bb-15b02ebb6f5d","Type":"ContainerDied","Data":"db46fc3bf6d0814a7e7d227a3141110ba8ea1f0ac6d031c66ca2a2318bfa2ee1"} Oct 31 04:00:26.023237 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:26.023132 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5" event={"ID":"531c8516-8a62-4add-a9bb-15b02ebb6f5d","Type":"ContainerDied","Data":"ffa10155a62575e746871081b0d464a9737378c1e97c64140ebe33a889587b96"} Oct 31 04:00:26.023237 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:26.023154 5490 scope.go:122] "RemoveContainer" containerID="db46fc3bf6d0814a7e7d227a3141110ba8ea1f0ac6d031c66ca2a2318bfa2ee1" Oct 31 04:00:26.030545 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:26.030508 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zfzn\" (UniqueName: \"kubernetes.io/projected/531c8516-8a62-4add-a9bb-15b02ebb6f5d-kube-api-access-7zfzn\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:26.030545 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:26.030533 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531c8516-8a62-4add-a9bb-15b02ebb6f5d-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:26.030545 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:26.030545 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/531c8516-8a62-4add-a9bb-15b02ebb6f5d-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:26.030851 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:26.030559 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/531c8516-8a62-4add-a9bb-15b02ebb6f5d-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:26.033971 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:26.033948 5490 scope.go:122] "RemoveContainer" containerID="db46fc3bf6d0814a7e7d227a3141110ba8ea1f0ac6d031c66ca2a2318bfa2ee1" Oct 31 04:00:26.034300 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:00:26.034278 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db46fc3bf6d0814a7e7d227a3141110ba8ea1f0ac6d031c66ca2a2318bfa2ee1\": container with ID starting with db46fc3bf6d0814a7e7d227a3141110ba8ea1f0ac6d031c66ca2a2318bfa2ee1 not found: ID does not exist" containerID="db46fc3bf6d0814a7e7d227a3141110ba8ea1f0ac6d031c66ca2a2318bfa2ee1" Oct 31 04:00:26.034377 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:26.034307 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db46fc3bf6d0814a7e7d227a3141110ba8ea1f0ac6d031c66ca2a2318bfa2ee1"} err="failed to get container status \"db46fc3bf6d0814a7e7d227a3141110ba8ea1f0ac6d031c66ca2a2318bfa2ee1\": rpc error: code = NotFound desc = could not find container \"db46fc3bf6d0814a7e7d227a3141110ba8ea1f0ac6d031c66ca2a2318bfa2ee1\": container with ID starting with db46fc3bf6d0814a7e7d227a3141110ba8ea1f0ac6d031c66ca2a2318bfa2ee1 not found: ID does not exist" Oct 31 04:00:26.051283 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:26.051191 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5"] Oct 31 04:00:26.055097 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:26.055072 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-bcc7a4-67b44f79d4-6trw5"] Oct 31 04:00:26.396476 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:26.396428 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:27.574057 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:27.574015 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="531c8516-8a62-4add-a9bb-15b02ebb6f5d" path="/var/lib/kubelet/pods/531c8516-8a62-4add-a9bb-15b02ebb6f5d/volumes" Oct 31 04:00:27.660238 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:27.660202 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:32.112176 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:32.112142 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:32.276166 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:32.276129 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:32.476305 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:32.476260 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:35.459146 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:35.459108 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-586ddd746f-nv8gm"] Oct 31 04:00:35.459592 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:35.459337 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-586ddd746f-nv8gm" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" containerID="cri-o://8d2b0c5a39c5ebe143891d27760c13a3b12577916375a95cdb0f7e959edac463" gracePeriod=30 Oct 31 04:00:35.715943 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:35.715875 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586ddd746f-nv8gm" Oct 31 04:00:35.822657 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:35.822623 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2238dc08-3a3e-426e-a8f7-22bec5f016c8-kube-api-access-bk745\" (UniqueName: \"kubernetes.io/projected/2238dc08-3a3e-426e-a8f7-22bec5f016c8-kube-api-access-bk745\") pod \"2238dc08-3a3e-426e-a8f7-22bec5f016c8\" (UID: \"2238dc08-3a3e-426e-a8f7-22bec5f016c8\") " Oct 31 04:00:35.822840 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:35.822742 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2238dc08-3a3e-426e-a8f7-22bec5f016c8-webhook-cert\" (UniqueName: \"kubernetes.io/secret/2238dc08-3a3e-426e-a8f7-22bec5f016c8-webhook-cert\") pod \"2238dc08-3a3e-426e-a8f7-22bec5f016c8\" (UID: \"2238dc08-3a3e-426e-a8f7-22bec5f016c8\") " Oct 31 04:00:35.822840 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:35.822809 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2238dc08-3a3e-426e-a8f7-22bec5f016c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2238dc08-3a3e-426e-a8f7-22bec5f016c8-metrics-certs\") pod \"2238dc08-3a3e-426e-a8f7-22bec5f016c8\" (UID: \"2238dc08-3a3e-426e-a8f7-22bec5f016c8\") " Oct 31 04:00:35.825302 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:35.825276 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2238dc08-3a3e-426e-a8f7-22bec5f016c8-webhook-cert" pod "2238dc08-3a3e-426e-a8f7-22bec5f016c8" (UID: "2238dc08-3a3e-426e-a8f7-22bec5f016c8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:00:35.826067 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:35.826042 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2238dc08-3a3e-426e-a8f7-22bec5f016c8-kube-api-access-bk745" pod "2238dc08-3a3e-426e-a8f7-22bec5f016c8" (UID: "2238dc08-3a3e-426e-a8f7-22bec5f016c8"). InnerVolumeSpecName "kube-api-access-bk745". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:00:35.826131 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:35.826046 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2238dc08-3a3e-426e-a8f7-22bec5f016c8-metrics-certs" pod "2238dc08-3a3e-426e-a8f7-22bec5f016c8" (UID: "2238dc08-3a3e-426e-a8f7-22bec5f016c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:00:35.924156 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:35.924121 5490 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2238dc08-3a3e-426e-a8f7-22bec5f016c8-webhook-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:35.924156 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:35.924151 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2238dc08-3a3e-426e-a8f7-22bec5f016c8-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:35.924156 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:35.924162 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk745\" (UniqueName: \"kubernetes.io/projected/2238dc08-3a3e-426e-a8f7-22bec5f016c8-kube-api-access-bk745\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:00:36.081001 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.080920 5490 generic.go:356] "Generic (PLEG): container finished" podID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerID="8d2b0c5a39c5ebe143891d27760c13a3b12577916375a95cdb0f7e959edac463" exitCode=0 Oct 31 04:00:36.081001 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.080973 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586ddd746f-nv8gm" Oct 31 04:00:36.081195 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.081007 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586ddd746f-nv8gm" event={"ID":"2238dc08-3a3e-426e-a8f7-22bec5f016c8","Type":"ContainerDied","Data":"8d2b0c5a39c5ebe143891d27760c13a3b12577916375a95cdb0f7e959edac463"} Oct 31 04:00:36.081195 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.081065 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586ddd746f-nv8gm" event={"ID":"2238dc08-3a3e-426e-a8f7-22bec5f016c8","Type":"ContainerDied","Data":"2f67fe9e6635eef884229986ab8bb74fe3eabb37e01e3a9657e6e4b202125138"} Oct 31 04:00:36.081195 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.081094 5490 scope.go:122] "RemoveContainer" containerID="8d2b0c5a39c5ebe143891d27760c13a3b12577916375a95cdb0f7e959edac463" Oct 31 04:00:36.091164 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.091143 5490 scope.go:122] "RemoveContainer" containerID="090b98d525cb749c2a0dbb244befddc00cca4ea40d6a6a013699eb4035ae57d8" Oct 31 04:00:36.108129 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.108102 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-586ddd746f-nv8gm"] Oct 31 04:00:36.110729 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.110703 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-586ddd746f-nv8gm"] Oct 31 04:00:36.112061 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.112044 5490 scope.go:122] "RemoveContainer" containerID="8d2b0c5a39c5ebe143891d27760c13a3b12577916375a95cdb0f7e959edac463" Oct 31 04:00:36.112376 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:00:36.112354 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d2b0c5a39c5ebe143891d27760c13a3b12577916375a95cdb0f7e959edac463\": container with ID starting with 8d2b0c5a39c5ebe143891d27760c13a3b12577916375a95cdb0f7e959edac463 not found: ID does not exist" containerID="8d2b0c5a39c5ebe143891d27760c13a3b12577916375a95cdb0f7e959edac463" Oct 31 04:00:36.112447 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.112383 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d2b0c5a39c5ebe143891d27760c13a3b12577916375a95cdb0f7e959edac463"} err="failed to get container status \"8d2b0c5a39c5ebe143891d27760c13a3b12577916375a95cdb0f7e959edac463\": rpc error: code = NotFound desc = could not find container \"8d2b0c5a39c5ebe143891d27760c13a3b12577916375a95cdb0f7e959edac463\": container with ID starting with 8d2b0c5a39c5ebe143891d27760c13a3b12577916375a95cdb0f7e959edac463 not found: ID does not exist" Oct 31 04:00:36.112447 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.112402 5490 scope.go:122] "RemoveContainer" containerID="090b98d525cb749c2a0dbb244befddc00cca4ea40d6a6a013699eb4035ae57d8" Oct 31 04:00:36.112696 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:00:36.112668 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"090b98d525cb749c2a0dbb244befddc00cca4ea40d6a6a013699eb4035ae57d8\": container with ID starting with 090b98d525cb749c2a0dbb244befddc00cca4ea40d6a6a013699eb4035ae57d8 not found: ID does not exist" containerID="090b98d525cb749c2a0dbb244befddc00cca4ea40d6a6a013699eb4035ae57d8" Oct 31 04:00:36.112749 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.112699 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090b98d525cb749c2a0dbb244befddc00cca4ea40d6a6a013699eb4035ae57d8"} err="failed to get container status \"090b98d525cb749c2a0dbb244befddc00cca4ea40d6a6a013699eb4035ae57d8\": rpc error: code = NotFound desc = could not find container \"090b98d525cb749c2a0dbb244befddc00cca4ea40d6a6a013699eb4035ae57d8\": container with ID starting with 090b98d525cb749c2a0dbb244befddc00cca4ea40d6a6a013699eb4035ae57d8 not found: ID does not exist" Oct 31 04:00:36.791129 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791093 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-586ddd746f-ftqn6"] Oct 31 04:00:36.791527 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791502 5490 cpu_manager.go:414] "containerMap: removing container" podUID="24a03b21-4c4f-422e-baa4-6e905b1b0bf5" containerName="istio-proxy" Oct 31 04:00:36.791527 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791512 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a03b21-4c4f-422e-baa4-6e905b1b0bf5" containerName="istio-proxy" Oct 31 04:00:36.791527 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791526 5490 cpu_manager.go:414] "containerMap: removing container" podUID="4cf50264-0d76-46ca-b2cb-998ae3bbeb1b" containerName="payload-processing" Oct 31 04:00:36.791637 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791531 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf50264-0d76-46ca-b2cb-998ae3bbeb1b" containerName="payload-processing" Oct 31 04:00:36.791637 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791546 5490 cpu_manager.go:414] "containerMap: removing container" podUID="76c026b4-65c7-4d57-9945-d4ebfb66ca70" containerName="maas-api" Oct 31 04:00:36.791637 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791553 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c026b4-65c7-4d57-9945-d4ebfb66ca70" containerName="maas-api" Oct 31 04:00:36.791637 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791562 5490 cpu_manager.go:414] "containerMap: removing container" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" Oct 31 04:00:36.791637 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791567 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" Oct 31 04:00:36.791637 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791574 5490 cpu_manager.go:414] "containerMap: removing container" podUID="1303047d-c782-41c7-bc35-d3298c9ef224" containerName="maas-api" Oct 31 04:00:36.791637 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791579 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1303047d-c782-41c7-bc35-d3298c9ef224" containerName="maas-api" Oct 31 04:00:36.791637 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791589 5490 cpu_manager.go:414] "containerMap: removing container" podUID="7d55416a-52be-41b0-aef9-7709346f1fdd" containerName="maas-api" Oct 31 04:00:36.791637 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791594 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d55416a-52be-41b0-aef9-7709346f1fdd" containerName="maas-api" Oct 31 04:00:36.791637 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791603 5490 cpu_manager.go:414] "containerMap: removing container" podUID="e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff" containerName="maas-api" Oct 31 04:00:36.791637 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791608 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff" containerName="maas-api" Oct 31 04:00:36.791637 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791615 5490 cpu_manager.go:414] "containerMap: removing container" podUID="679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2" containerName="payload-pre-processing" Oct 31 04:00:36.791637 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791620 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2" containerName="payload-pre-processing" Oct 31 04:00:36.791637 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791631 5490 cpu_manager.go:414] "containerMap: removing container" podUID="42e53881-926d-482d-9557-a86d3ae3f9d9" containerName="payload-pre-processing" Oct 31 04:00:36.791637 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791635 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42e53881-926d-482d-9557-a86d3ae3f9d9" containerName="payload-pre-processing" Oct 31 04:00:36.791637 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791643 5490 cpu_manager.go:414] "containerMap: removing container" podUID="2cf184db-500a-41eb-8ca5-7a40b2645693" containerName="payload-processing" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791648 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf184db-500a-41eb-8ca5-7a40b2645693" containerName="payload-processing" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791660 5490 cpu_manager.go:414] "containerMap: removing container" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791665 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791691 5490 cpu_manager.go:414] "containerMap: removing container" podUID="531c8516-8a62-4add-a9bb-15b02ebb6f5d" containerName="maas-api" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791696 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="531c8516-8a62-4add-a9bb-15b02ebb6f5d" containerName="maas-api" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791702 5490 cpu_manager.go:414] "containerMap: removing container" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791706 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791713 5490 cpu_manager.go:414] "containerMap: removing container" podUID="a8a2a69d-e4b7-4338-a7b4-22042b94432f" containerName="istio-proxy" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791717 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a2a69d-e4b7-4338-a7b4-22042b94432f" containerName="istio-proxy" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791729 5490 cpu_manager.go:414] "containerMap: removing container" podUID="b6b9fc40-2379-498d-84f5-ce5550fd76ae" containerName="istio-proxy" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791733 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b9fc40-2379-498d-84f5-ce5550fd76ae" containerName="istio-proxy" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791742 5490 cpu_manager.go:414] "containerMap: removing container" podUID="a8a2a69d-e4b7-4338-a7b4-22042b94432f" containerName="istio-proxy" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791746 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a2a69d-e4b7-4338-a7b4-22042b94432f" containerName="istio-proxy" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791807 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a03b21-4c4f-422e-baa4-6e905b1b0bf5" containerName="istio-proxy" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791816 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf50264-0d76-46ca-b2cb-998ae3bbeb1b" containerName="payload-processing" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791824 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c026b4-65c7-4d57-9945-d4ebfb66ca70" containerName="maas-api" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791831 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791836 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="1303047d-c782-41c7-bc35-d3298c9ef224" containerName="maas-api" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791843 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d55416a-52be-41b0-aef9-7709346f1fdd" containerName="maas-api" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791848 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c359bb-eb6c-4fc6-8a49-86e4d4ca43ff" containerName="maas-api" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791853 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="679e88b7-c835-4f3b-bb3b-4aa5a1fcefe2" containerName="payload-pre-processing" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791860 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="42e53881-926d-482d-9557-a86d3ae3f9d9" containerName="payload-pre-processing" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791867 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf184db-500a-41eb-8ca5-7a40b2645693" containerName="payload-processing" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791874 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="531c8516-8a62-4add-a9bb-15b02ebb6f5d" containerName="maas-api" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791880 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="1712a46e-a034-40f6-a276-09a907dd082f" containerName="main" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791886 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a2a69d-e4b7-4338-a7b4-22042b94432f" containerName="istio-proxy" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791893 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b9fc40-2379-498d-84f5-ce5550fd76ae" containerName="istio-proxy" Oct 31 04:00:36.792211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.791898 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a2a69d-e4b7-4338-a7b4-22042b94432f" containerName="istio-proxy" Oct 31 04:00:36.796737 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.796715 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586ddd746f-ftqn6" Oct 31 04:00:36.802029 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.802000 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 04:00:36.802952 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.802850 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 04:00:36.802952 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.802910 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cjzft\"" Oct 31 04:00:36.803410 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.803284 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 04:00:36.806656 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.806632 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-586ddd746f-ftqn6"] Oct 31 04:00:36.934936 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.934896 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a8a82efe-a659-46ac-a33f-87679ba636bc-webhook-cert\") pod \"maas-controller-586ddd746f-ftqn6\" (UID: \"a8a82efe-a659-46ac-a33f-87679ba636bc\") " pod="opendatahub/maas-controller-586ddd746f-ftqn6" Oct 31 04:00:36.934936 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.934941 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a82efe-a659-46ac-a33f-87679ba636bc-metrics-certs\") pod \"maas-controller-586ddd746f-ftqn6\" (UID: \"a8a82efe-a659-46ac-a33f-87679ba636bc\") " pod="opendatahub/maas-controller-586ddd746f-ftqn6" Oct 31 04:00:36.935185 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:36.935018 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p87q\" (UniqueName: \"kubernetes.io/projected/a8a82efe-a659-46ac-a33f-87679ba636bc-kube-api-access-9p87q\") pod \"maas-controller-586ddd746f-ftqn6\" (UID: \"a8a82efe-a659-46ac-a33f-87679ba636bc\") " pod="opendatahub/maas-controller-586ddd746f-ftqn6" Oct 31 04:00:37.036070 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:37.036032 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a8a82efe-a659-46ac-a33f-87679ba636bc-webhook-cert\") pod \"maas-controller-586ddd746f-ftqn6\" (UID: \"a8a82efe-a659-46ac-a33f-87679ba636bc\") " pod="opendatahub/maas-controller-586ddd746f-ftqn6" Oct 31 04:00:37.036070 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:37.036078 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a82efe-a659-46ac-a33f-87679ba636bc-metrics-certs\") pod \"maas-controller-586ddd746f-ftqn6\" (UID: \"a8a82efe-a659-46ac-a33f-87679ba636bc\") " pod="opendatahub/maas-controller-586ddd746f-ftqn6" Oct 31 04:00:37.036329 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:37.036102 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p87q\" (UniqueName: \"kubernetes.io/projected/a8a82efe-a659-46ac-a33f-87679ba636bc-kube-api-access-9p87q\") pod \"maas-controller-586ddd746f-ftqn6\" (UID: \"a8a82efe-a659-46ac-a33f-87679ba636bc\") " pod="opendatahub/maas-controller-586ddd746f-ftqn6" Oct 31 04:00:37.038655 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:37.038634 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a8a82efe-a659-46ac-a33f-87679ba636bc-webhook-cert\") pod \"maas-controller-586ddd746f-ftqn6\" (UID: \"a8a82efe-a659-46ac-a33f-87679ba636bc\") " pod="opendatahub/maas-controller-586ddd746f-ftqn6" Oct 31 04:00:37.038829 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:37.038804 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a82efe-a659-46ac-a33f-87679ba636bc-metrics-certs\") pod \"maas-controller-586ddd746f-ftqn6\" (UID: \"a8a82efe-a659-46ac-a33f-87679ba636bc\") " pod="opendatahub/maas-controller-586ddd746f-ftqn6" Oct 31 04:00:37.044474 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:37.044415 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p87q\" (UniqueName: \"kubernetes.io/projected/a8a82efe-a659-46ac-a33f-87679ba636bc-kube-api-access-9p87q\") pod \"maas-controller-586ddd746f-ftqn6\" (UID: \"a8a82efe-a659-46ac-a33f-87679ba636bc\") " pod="opendatahub/maas-controller-586ddd746f-ftqn6" Oct 31 04:00:37.112496 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:37.112464 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586ddd746f-ftqn6" Oct 31 04:00:37.390973 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:37.390945 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-586ddd746f-ftqn6"] Oct 31 04:00:37.393969 ip-10-0-142-76 kubenswrapper[5490]: W1031 04:00:37.393937 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a82efe_a659_46ac_a33f_87679ba636bc.slice/crio-183a87e2b48b7828e07b5470ee6a86e4977048565f8bb9ddc2684a8175de6922 WatchSource:0}: Error finding container 183a87e2b48b7828e07b5470ee6a86e4977048565f8bb9ddc2684a8175de6922: Status 404 returned error can't find the container with id 183a87e2b48b7828e07b5470ee6a86e4977048565f8bb9ddc2684a8175de6922 Oct 31 04:00:37.395926 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:37.395901 5490 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:00:37.576664 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:37.576625 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" path="/var/lib/kubelet/pods/2238dc08-3a3e-426e-a8f7-22bec5f016c8/volumes" Oct 31 04:00:38.095671 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:38.095625 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586ddd746f-ftqn6" event={"ID":"a8a82efe-a659-46ac-a33f-87679ba636bc","Type":"ContainerStarted","Data":"332049e7346f32b30d35f6ad0d92b483b89ec7d1a9352b7191fbe7c16a30268f"} Oct 31 04:00:38.096115 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:38.095705 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586ddd746f-ftqn6" event={"ID":"a8a82efe-a659-46ac-a33f-87679ba636bc","Type":"ContainerStarted","Data":"183a87e2b48b7828e07b5470ee6a86e4977048565f8bb9ddc2684a8175de6922"} Oct 31 04:00:38.096115 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:38.095768 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-586ddd746f-ftqn6" Oct 31 04:00:38.127581 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:38.127520 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-586ddd746f-ftqn6" podStartSLOduration=1.557922256 podStartE2EDuration="2.127505384s" totalImagesPullingTime="569.583128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:38.125304369 +0000 UTC m=+2672.659999325" watchObservedRunningTime="2026-10-31 04:00:38.127505384 +0000 UTC m=+2672.662200339" Oct 31 04:00:41.909551 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:41.909505 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:42.039620 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:42.039580 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:42.328987 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:42.328897 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:47.127412 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:00:47.127374 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:00:47.862037 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:47.861996 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:00:49.104624 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:00:49.104586 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-586ddd746f-ftqn6" Oct 31 04:01:00.961369 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:00.961329 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8"] Oct 31 04:01:00.963936 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:00.961796 5490 cpu_manager.go:414] "containerMap: removing container" podUID="24a03b21-4c4f-422e-baa4-6e905b1b0bf5" containerName="istio-proxy" Oct 31 04:01:00.963936 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:00.961810 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a03b21-4c4f-422e-baa4-6e905b1b0bf5" containerName="istio-proxy" Oct 31 04:01:00.963936 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:00.961913 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="2238dc08-3a3e-426e-a8f7-22bec5f016c8" containerName="manager" Oct 31 04:01:00.963936 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:00.961929 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a03b21-4c4f-422e-baa4-6e905b1b0bf5" containerName="istio-proxy" Oct 31 04:01:00.965080 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:00.965057 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:00.977497 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:00.977467 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8"] Oct 31 04:01:01.066406 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.066363 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.066406 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.066403 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.066783 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.066422 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.066783 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.066491 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.066783 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.066551 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.066783 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.066669 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg868\" (UniqueName: \"kubernetes.io/projected/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-kube-api-access-jg868\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.066783 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.066767 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.066996 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.066796 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.066996 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.066820 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.167267 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.167223 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg868\" (UniqueName: \"kubernetes.io/projected/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-kube-api-access-jg868\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.167267 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.167269 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.167557 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.167305 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.167557 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.167330 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.167557 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.167357 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.167557 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.167376 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.167557 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.167393 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.167557 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.167409 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.167946 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.167642 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.167946 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.167928 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.168054 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.167984 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.168127 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.168057 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.168127 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.168073 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.168221 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.168199 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.169916 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.169889 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.170165 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.170145 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.175435 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.175413 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.175656 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.175638 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg868\" (UniqueName: \"kubernetes.io/projected/8eae7b8f-98ca-4de7-92e0-ff47b28d6927-kube-api-access-jg868\") pod \"maas-default-gateway-openshift-default-769df7846c-rssq8\" (UID: \"8eae7b8f-98ca-4de7-92e0-ff47b28d6927\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.282400 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.282308 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:01.563517 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.563490 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8"] Oct 31 04:01:01.565283 ip-10-0-142-76 kubenswrapper[5490]: W1031 04:01:01.565257 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eae7b8f_98ca_4de7_92e0_ff47b28d6927.slice/crio-e86d489f812a4e837236dae5dc59f3cdd89ab817f47a3efc8fc9a86fd27004de WatchSource:0}: Error finding container e86d489f812a4e837236dae5dc59f3cdd89ab817f47a3efc8fc9a86fd27004de: Status 404 returned error can't find the container with id e86d489f812a4e837236dae5dc59f3cdd89ab817f47a3efc8fc9a86fd27004de Oct 31 04:01:01.568396 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.568362 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:01:01.568504 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.568427 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:01:01.568504 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:01.568458 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:01:02.234645 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:02.234599 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" event={"ID":"8eae7b8f-98ca-4de7-92e0-ff47b28d6927","Type":"ContainerStarted","Data":"05da12ecefba7a4d5fb2a11284ca032731fd9c3b11151b06bb4f8cc9fb997324"} Oct 31 04:01:02.235318 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:02.235294 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" event={"ID":"8eae7b8f-98ca-4de7-92e0-ff47b28d6927","Type":"ContainerStarted","Data":"e86d489f812a4e837236dae5dc59f3cdd89ab817f47a3efc8fc9a86fd27004de"} Oct 31 04:01:02.267953 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:02.267897 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" podStartSLOduration=2.267882852 podStartE2EDuration="2.267882852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:01:02.266743415 +0000 UTC m=+2696.801438370" watchObservedRunningTime="2026-10-31 04:01:02.267882852 +0000 UTC m=+2696.802577807" Oct 31 04:01:02.283133 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:02.283099 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:02.284665 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:02.284638 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rssq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 31 04:01:02.284806 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:02.284697 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" podUID="8eae7b8f-98ca-4de7-92e0-ff47b28d6927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 31 04:01:03.283460 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:03.283419 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rssq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 31 04:01:03.284101 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:03.283485 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" podUID="8eae7b8f-98ca-4de7-92e0-ff47b28d6927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 31 04:01:04.283704 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:04.283658 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rssq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 31 04:01:04.284128 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:04.283724 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" podUID="8eae7b8f-98ca-4de7-92e0-ff47b28d6927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 31 04:01:05.283666 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:05.283630 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rssq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 31 04:01:05.283871 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:05.283718 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" podUID="8eae7b8f-98ca-4de7-92e0-ff47b28d6927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 31 04:01:06.283250 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:06.283211 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rssq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 31 04:01:06.283456 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:06.283277 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" podUID="8eae7b8f-98ca-4de7-92e0-ff47b28d6927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 31 04:01:06.747703 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:06.747289 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:06.855561 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:06.855510 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-586ddd746f-ftqn6"] Oct 31 04:01:06.856571 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:06.856538 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-586ddd746f-ftqn6" podUID="a8a82efe-a659-46ac-a33f-87679ba636bc" containerName="manager" containerID="cri-o://332049e7346f32b30d35f6ad0d92b483b89ec7d1a9352b7191fbe7c16a30268f" gracePeriod=30 Oct 31 04:01:07.182578 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.182544 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586ddd746f-ftqn6" Oct 31 04:01:07.247703 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.236485 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a82efe-a659-46ac-a33f-87679ba636bc-kube-api-access-9p87q\" (UniqueName: \"kubernetes.io/projected/a8a82efe-a659-46ac-a33f-87679ba636bc-kube-api-access-9p87q\") pod \"a8a82efe-a659-46ac-a33f-87679ba636bc\" (UID: \"a8a82efe-a659-46ac-a33f-87679ba636bc\") " Oct 31 04:01:07.247703 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.236564 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8a82efe-a659-46ac-a33f-87679ba636bc-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a8a82efe-a659-46ac-a33f-87679ba636bc-webhook-cert\") pod \"a8a82efe-a659-46ac-a33f-87679ba636bc\" (UID: \"a8a82efe-a659-46ac-a33f-87679ba636bc\") " Oct 31 04:01:07.247703 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.236613 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8a82efe-a659-46ac-a33f-87679ba636bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a82efe-a659-46ac-a33f-87679ba636bc-metrics-certs\") pod \"a8a82efe-a659-46ac-a33f-87679ba636bc\" (UID: \"a8a82efe-a659-46ac-a33f-87679ba636bc\") " Oct 31 04:01:07.258228 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.258179 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a82efe-a659-46ac-a33f-87679ba636bc-metrics-certs" pod "a8a82efe-a659-46ac-a33f-87679ba636bc" (UID: "a8a82efe-a659-46ac-a33f-87679ba636bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:01:07.262857 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.262818 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a82efe-a659-46ac-a33f-87679ba636bc-kube-api-access-9p87q" pod "a8a82efe-a659-46ac-a33f-87679ba636bc" (UID: "a8a82efe-a659-46ac-a33f-87679ba636bc"). InnerVolumeSpecName "kube-api-access-9p87q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:01:07.262857 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.262818 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a82efe-a659-46ac-a33f-87679ba636bc-webhook-cert" pod "a8a82efe-a659-46ac-a33f-87679ba636bc" (UID: "a8a82efe-a659-46ac-a33f-87679ba636bc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:01:07.281566 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.281527 5490 generic.go:356] "Generic (PLEG): container finished" podID="a8a82efe-a659-46ac-a33f-87679ba636bc" containerID="332049e7346f32b30d35f6ad0d92b483b89ec7d1a9352b7191fbe7c16a30268f" exitCode=0 Oct 31 04:01:07.281799 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.281774 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586ddd746f-ftqn6" event={"ID":"a8a82efe-a659-46ac-a33f-87679ba636bc","Type":"ContainerDied","Data":"332049e7346f32b30d35f6ad0d92b483b89ec7d1a9352b7191fbe7c16a30268f"} Oct 31 04:01:07.281872 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.281852 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586ddd746f-ftqn6" event={"ID":"a8a82efe-a659-46ac-a33f-87679ba636bc","Type":"ContainerDied","Data":"183a87e2b48b7828e07b5470ee6a86e4977048565f8bb9ddc2684a8175de6922"} Oct 31 04:01:07.281912 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.281888 5490 scope.go:122] "RemoveContainer" containerID="332049e7346f32b30d35f6ad0d92b483b89ec7d1a9352b7191fbe7c16a30268f" Oct 31 04:01:07.282084 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.282066 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586ddd746f-ftqn6" Oct 31 04:01:07.283290 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.283262 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rssq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 31 04:01:07.283398 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.283307 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" podUID="8eae7b8f-98ca-4de7-92e0-ff47b28d6927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 31 04:01:07.292184 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.292100 5490 scope.go:122] "RemoveContainer" containerID="332049e7346f32b30d35f6ad0d92b483b89ec7d1a9352b7191fbe7c16a30268f" Oct 31 04:01:07.292390 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:01:07.292367 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"332049e7346f32b30d35f6ad0d92b483b89ec7d1a9352b7191fbe7c16a30268f\": container with ID starting with 332049e7346f32b30d35f6ad0d92b483b89ec7d1a9352b7191fbe7c16a30268f not found: ID does not exist" containerID="332049e7346f32b30d35f6ad0d92b483b89ec7d1a9352b7191fbe7c16a30268f" Oct 31 04:01:07.292465 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.292402 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"332049e7346f32b30d35f6ad0d92b483b89ec7d1a9352b7191fbe7c16a30268f"} err="failed to get container status \"332049e7346f32b30d35f6ad0d92b483b89ec7d1a9352b7191fbe7c16a30268f\": rpc error: code = NotFound desc = could not find container \"332049e7346f32b30d35f6ad0d92b483b89ec7d1a9352b7191fbe7c16a30268f\": container with ID starting with 332049e7346f32b30d35f6ad0d92b483b89ec7d1a9352b7191fbe7c16a30268f not found: ID does not exist" Oct 31 04:01:07.309705 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.309648 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-586ddd746f-ftqn6"] Oct 31 04:01:07.313625 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.313565 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-586ddd746f-ftqn6"] Oct 31 04:01:07.356923 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.339097 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p87q\" (UniqueName: \"kubernetes.io/projected/a8a82efe-a659-46ac-a33f-87679ba636bc-kube-api-access-9p87q\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:01:07.356923 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.339138 5490 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a8a82efe-a659-46ac-a33f-87679ba636bc-webhook-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:01:07.356923 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.339155 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a82efe-a659-46ac-a33f-87679ba636bc-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:01:07.577532 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:07.577446 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8a82efe-a659-46ac-a33f-87679ba636bc" path="/var/lib/kubelet/pods/a8a82efe-a659-46ac-a33f-87679ba636bc/volumes" Oct 31 04:01:08.283469 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.283424 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rssq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 31 04:01:08.284013 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.283482 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" podUID="8eae7b8f-98ca-4de7-92e0-ff47b28d6927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 31 04:01:08.402590 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.402547 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:08.506725 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.506668 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-586ddd746f-hdwdt"] Oct 31 04:01:08.507392 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.507370 5490 cpu_manager.go:414] "containerMap: removing container" podUID="a8a82efe-a659-46ac-a33f-87679ba636bc" containerName="manager" Oct 31 04:01:08.507489 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.507396 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a82efe-a659-46ac-a33f-87679ba636bc" containerName="manager" Oct 31 04:01:08.507551 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.507528 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a82efe-a659-46ac-a33f-87679ba636bc" containerName="manager" Oct 31 04:01:08.513068 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.513037 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586ddd746f-hdwdt" Oct 31 04:01:08.517280 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.517248 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 04:01:08.517437 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.517288 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 04:01:08.517437 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.517362 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-cjzft\"" Oct 31 04:01:08.517437 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.517428 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 04:01:08.519932 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.519908 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-586ddd746f-hdwdt"] Oct 31 04:01:08.552173 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.552073 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f21a691b-8bb1-480d-8d90-d6b78aac8624-webhook-cert\") pod \"maas-controller-586ddd746f-hdwdt\" (UID: \"f21a691b-8bb1-480d-8d90-d6b78aac8624\") " pod="opendatahub/maas-controller-586ddd746f-hdwdt" Oct 31 04:01:08.552173 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.552132 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f21a691b-8bb1-480d-8d90-d6b78aac8624-metrics-certs\") pod \"maas-controller-586ddd746f-hdwdt\" (UID: \"f21a691b-8bb1-480d-8d90-d6b78aac8624\") " pod="opendatahub/maas-controller-586ddd746f-hdwdt" Oct 31 04:01:08.552428 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.552317 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq5r6\" (UniqueName: \"kubernetes.io/projected/f21a691b-8bb1-480d-8d90-d6b78aac8624-kube-api-access-tq5r6\") pod \"maas-controller-586ddd746f-hdwdt\" (UID: \"f21a691b-8bb1-480d-8d90-d6b78aac8624\") " pod="opendatahub/maas-controller-586ddd746f-hdwdt" Oct 31 04:01:08.653155 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.653117 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f21a691b-8bb1-480d-8d90-d6b78aac8624-webhook-cert\") pod \"maas-controller-586ddd746f-hdwdt\" (UID: \"f21a691b-8bb1-480d-8d90-d6b78aac8624\") " pod="opendatahub/maas-controller-586ddd746f-hdwdt" Oct 31 04:01:08.653155 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.653158 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f21a691b-8bb1-480d-8d90-d6b78aac8624-metrics-certs\") pod \"maas-controller-586ddd746f-hdwdt\" (UID: \"f21a691b-8bb1-480d-8d90-d6b78aac8624\") " pod="opendatahub/maas-controller-586ddd746f-hdwdt" Oct 31 04:01:08.653417 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.653214 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq5r6\" (UniqueName: \"kubernetes.io/projected/f21a691b-8bb1-480d-8d90-d6b78aac8624-kube-api-access-tq5r6\") pod \"maas-controller-586ddd746f-hdwdt\" (UID: \"f21a691b-8bb1-480d-8d90-d6b78aac8624\") " pod="opendatahub/maas-controller-586ddd746f-hdwdt" Oct 31 04:01:08.655876 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.655841 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f21a691b-8bb1-480d-8d90-d6b78aac8624-metrics-certs\") pod \"maas-controller-586ddd746f-hdwdt\" (UID: \"f21a691b-8bb1-480d-8d90-d6b78aac8624\") " pod="opendatahub/maas-controller-586ddd746f-hdwdt" Oct 31 04:01:08.656011 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.655842 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f21a691b-8bb1-480d-8d90-d6b78aac8624-webhook-cert\") pod \"maas-controller-586ddd746f-hdwdt\" (UID: \"f21a691b-8bb1-480d-8d90-d6b78aac8624\") " pod="opendatahub/maas-controller-586ddd746f-hdwdt" Oct 31 04:01:08.661252 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.661226 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq5r6\" (UniqueName: \"kubernetes.io/projected/f21a691b-8bb1-480d-8d90-d6b78aac8624-kube-api-access-tq5r6\") pod \"maas-controller-586ddd746f-hdwdt\" (UID: \"f21a691b-8bb1-480d-8d90-d6b78aac8624\") " pod="opendatahub/maas-controller-586ddd746f-hdwdt" Oct 31 04:01:08.831544 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:08.831445 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586ddd746f-hdwdt" Oct 31 04:01:09.114268 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:09.114239 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-586ddd746f-hdwdt"] Oct 31 04:01:09.116208 ip-10-0-142-76 kubenswrapper[5490]: W1031 04:01:09.116175 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21a691b_8bb1_480d_8d90_d6b78aac8624.slice/crio-1d1f2411a13a9458b200b1ca3bf090470f2e346be5107c8b89fddad6f9e16d3d WatchSource:0}: Error finding container 1d1f2411a13a9458b200b1ca3bf090470f2e346be5107c8b89fddad6f9e16d3d: Status 404 returned error can't find the container with id 1d1f2411a13a9458b200b1ca3bf090470f2e346be5107c8b89fddad6f9e16d3d Oct 31 04:01:09.296413 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:09.296375 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586ddd746f-hdwdt" event={"ID":"f21a691b-8bb1-480d-8d90-d6b78aac8624","Type":"ContainerStarted","Data":"1d1f2411a13a9458b200b1ca3bf090470f2e346be5107c8b89fddad6f9e16d3d"} Oct 31 04:01:09.840476 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:09.840388 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:09.841238 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:09.841202 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:09.842069 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:09.842044 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" Oct 31 04:01:10.303279 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:10.303243 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586ddd746f-hdwdt" event={"ID":"f21a691b-8bb1-480d-8d90-d6b78aac8624","Type":"ContainerStarted","Data":"d7eed6a87104d315a4e3258fda72e5b5eeabe5a0a4e9096f12428f69251b4ead"} Oct 31 04:01:10.303840 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:10.303336 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-586ddd746f-hdwdt" Oct 31 04:01:10.320670 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:10.320622 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-586ddd746f-hdwdt" podStartSLOduration=1.8923837940000001 podStartE2EDuration="2.320597316s" totalImagesPullingTime="428.213522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:01:10.318983434 +0000 UTC m=+2704.853678389" watchObservedRunningTime="2026-10-31 04:01:10.320597316 +0000 UTC m=+2704.855292271" Oct 31 04:01:16.037828 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.037744 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd"] Oct 31 04:01:16.041550 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.041529 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.055625 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.055596 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd"] Oct 31 04:01:16.127899 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.127857 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd4jr\" (UniqueName: \"kubernetes.io/projected/b3c2974d-83ad-4321-93eb-60ec9738d588-kube-api-access-dd4jr\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.128069 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.127977 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3c2974d-83ad-4321-93eb-60ec9738d588-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.128069 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.128023 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c2974d-83ad-4321-93eb-60ec9738d588-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.128069 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.128048 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c2974d-83ad-4321-93eb-60ec9738d588-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.128211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.128137 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c2974d-83ad-4321-93eb-60ec9738d588-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.128211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.128186 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c2974d-83ad-4321-93eb-60ec9738d588-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.128284 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.128222 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c2974d-83ad-4321-93eb-60ec9738d588-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.128284 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.128274 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c2974d-83ad-4321-93eb-60ec9738d588-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.128396 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.128331 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c2974d-83ad-4321-93eb-60ec9738d588-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.229190 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.229145 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c2974d-83ad-4321-93eb-60ec9738d588-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.229190 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.229194 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c2974d-83ad-4321-93eb-60ec9738d588-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.229435 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.229255 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd4jr\" (UniqueName: \"kubernetes.io/projected/b3c2974d-83ad-4321-93eb-60ec9738d588-kube-api-access-dd4jr\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.229435 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.229320 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3c2974d-83ad-4321-93eb-60ec9738d588-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.229435 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.229368 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c2974d-83ad-4321-93eb-60ec9738d588-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.229435 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.229398 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c2974d-83ad-4321-93eb-60ec9738d588-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.229647 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.229445 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c2974d-83ad-4321-93eb-60ec9738d588-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.229647 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.229473 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c2974d-83ad-4321-93eb-60ec9738d588-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.229647 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.229510 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c2974d-83ad-4321-93eb-60ec9738d588-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.229868 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.229786 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3c2974d-83ad-4321-93eb-60ec9738d588-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.229919 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.229873 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3c2974d-83ad-4321-93eb-60ec9738d588-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.229919 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.229887 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c2974d-83ad-4321-93eb-60ec9738d588-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.229995 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.229941 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3c2974d-83ad-4321-93eb-60ec9738d588-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.230196 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.230174 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3c2974d-83ad-4321-93eb-60ec9738d588-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.231633 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.231602 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3c2974d-83ad-4321-93eb-60ec9738d588-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.232154 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.232135 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3c2974d-83ad-4321-93eb-60ec9738d588-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.237731 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.237699 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3c2974d-83ad-4321-93eb-60ec9738d588-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.237916 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.237901 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd4jr\" (UniqueName: \"kubernetes.io/projected/b3c2974d-83ad-4321-93eb-60ec9738d588-kube-api-access-dd4jr\") pod \"maas-default-gateway-openshift-default-769df7846c-svnrd\" (UID: \"b3c2974d-83ad-4321-93eb-60ec9738d588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.356514 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.356429 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:16.635818 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.635789 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd"] Oct 31 04:01:16.637531 ip-10-0-142-76 kubenswrapper[5490]: W1031 04:01:16.637499 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c2974d_83ad_4321_93eb_60ec9738d588.slice/crio-cbcb45953596f7b4d075079d7f8e8616a5e2fc171df822006d1ca12f53f3bb4e WatchSource:0}: Error finding container cbcb45953596f7b4d075079d7f8e8616a5e2fc171df822006d1ca12f53f3bb4e: Status 404 returned error can't find the container with id cbcb45953596f7b4d075079d7f8e8616a5e2fc171df822006d1ca12f53f3bb4e Oct 31 04:01:16.639770 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.639740 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:01:16.639849 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.639802 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:01:16.639849 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:16.639832 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:01:17.150708 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:01:17.150349 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:01:17.365169 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:17.365132 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" event={"ID":"b3c2974d-83ad-4321-93eb-60ec9738d588","Type":"ContainerStarted","Data":"351f9e47364a1a80386c8a5648b3312071797962b76e3bed3a003b09dc73869c"} Oct 31 04:01:17.365358 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:17.365194 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" event={"ID":"b3c2974d-83ad-4321-93eb-60ec9738d588","Type":"ContainerStarted","Data":"cbcb45953596f7b4d075079d7f8e8616a5e2fc171df822006d1ca12f53f3bb4e"} Oct 31 04:01:17.388109 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:17.388057 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" podStartSLOduration=1.388029336 podStartE2EDuration="1.388029336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:01:17.384995461 +0000 UTC m=+2711.919690417" watchObservedRunningTime="2026-10-31 04:01:17.388029336 +0000 UTC m=+2711.922724291" Oct 31 04:01:18.356585 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:18.356555 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:18.358376 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:18.358347 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-svnrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 04:01:18.358517 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:18.358397 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" podUID="b3c2974d-83ad-4321-93eb-60ec9738d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 04:01:18.591913 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:18.591869 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:19.357204 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:19.357166 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-svnrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 04:01:19.357611 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:19.357226 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" podUID="b3c2974d-83ad-4321-93eb-60ec9738d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 04:01:20.358054 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:20.358008 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-svnrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 04:01:20.358565 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:20.358104 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" podUID="b3c2974d-83ad-4321-93eb-60ec9738d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 04:01:21.311664 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:21.311630 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-586ddd746f-hdwdt" Oct 31 04:01:21.357299 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:21.357256 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-svnrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 04:01:21.357546 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:21.357319 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" podUID="b3c2974d-83ad-4321-93eb-60ec9738d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 04:01:22.357258 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:22.357216 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-svnrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 04:01:22.357657 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:22.357304 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" podUID="b3c2974d-83ad-4321-93eb-60ec9738d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 04:01:23.357898 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:23.357857 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-svnrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 31 04:01:23.358314 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:23.357930 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" podUID="b3c2974d-83ad-4321-93eb-60ec9738d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 31 04:01:24.901513 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:24.901474 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:24.902070 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:24.901957 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:24.902915 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:24.902891 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" Oct 31 04:01:33.358632 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:33.358592 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:33.497173 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:33.497092 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:01:40.149382 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.149345 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m"] Oct 31 04:01:40.150027 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.149585 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" podUID="decb2697-571c-48e2-88b3-84a03dfbd80a" containerName="discovery" containerID="cri-o://be2efe71c5d89dd064bd84b0cbaeb4f452545b619be5ec4cbc00703b6e0dc887" gracePeriod=30 Oct 31 04:01:40.421087 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.421051 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 04:01:40.492481 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.492443 5490 generic.go:356] "Generic (PLEG): container finished" podID="decb2697-571c-48e2-88b3-84a03dfbd80a" containerID="be2efe71c5d89dd064bd84b0cbaeb4f452545b619be5ec4cbc00703b6e0dc887" exitCode=0 Oct 31 04:01:40.492705 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.492502 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" Oct 31 04:01:40.492705 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.492503 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" event={"ID":"decb2697-571c-48e2-88b3-84a03dfbd80a","Type":"ContainerDied","Data":"be2efe71c5d89dd064bd84b0cbaeb4f452545b619be5ec4cbc00703b6e0dc887"} Oct 31 04:01:40.492705 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.492620 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m" event={"ID":"decb2697-571c-48e2-88b3-84a03dfbd80a","Type":"ContainerDied","Data":"ac24b9074865b1e8b02dd6467baa265e601bd129deb3bf2fff9a64acaf1cf13a"} Oct 31 04:01:40.492705 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.492641 5490 scope.go:122] "RemoveContainer" containerID="be2efe71c5d89dd064bd84b0cbaeb4f452545b619be5ec4cbc00703b6e0dc887" Oct 31 04:01:40.499125 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.499095 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-token\" (UniqueName: \"kubernetes.io/projected/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-token\") pod \"decb2697-571c-48e2-88b3-84a03dfbd80a\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " Oct 31 04:01:40.499261 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.499150 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-kubeconfig\") pod \"decb2697-571c-48e2-88b3-84a03dfbd80a\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " Oct 31 04:01:40.499261 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.499186 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-csr-ca-configmap\") pod \"decb2697-571c-48e2-88b3-84a03dfbd80a\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " Oct 31 04:01:40.499261 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.499217 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/decb2697-571c-48e2-88b3-84a03dfbd80a-kube-api-access-2g8sv\" (UniqueName: \"kubernetes.io/projected/decb2697-571c-48e2-88b3-84a03dfbd80a-kube-api-access-2g8sv\") pod \"decb2697-571c-48e2-88b3-84a03dfbd80a\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " Oct 31 04:01:40.499455 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.499272 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/decb2697-571c-48e2-88b3-84a03dfbd80a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/decb2697-571c-48e2-88b3-84a03dfbd80a-local-certs\") pod \"decb2697-571c-48e2-88b3-84a03dfbd80a\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " Oct 31 04:01:40.499455 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.499310 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-cacerts\" (UniqueName: \"kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-cacerts\") pod \"decb2697-571c-48e2-88b3-84a03dfbd80a\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " Oct 31 04:01:40.499455 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.499362 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-csr-dns-cert\") pod \"decb2697-571c-48e2-88b3-84a03dfbd80a\" (UID: \"decb2697-571c-48e2-88b3-84a03dfbd80a\") " Oct 31 04:01:40.499724 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.499693 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-csr-ca-configmap" pod "decb2697-571c-48e2-88b3-84a03dfbd80a" (UID: "decb2697-571c-48e2-88b3-84a03dfbd80a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:01:40.499860 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.499806 5490 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-csr-ca-configmap\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:01:40.501807 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.501778 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-cacerts" pod "decb2697-571c-48e2-88b3-84a03dfbd80a" (UID: "decb2697-571c-48e2-88b3-84a03dfbd80a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:01:40.502100 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.502071 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/decb2697-571c-48e2-88b3-84a03dfbd80a-kube-api-access-2g8sv" pod "decb2697-571c-48e2-88b3-84a03dfbd80a" (UID: "decb2697-571c-48e2-88b3-84a03dfbd80a"). InnerVolumeSpecName "kube-api-access-2g8sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:01:40.502198 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.502165 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-kubeconfig" pod "decb2697-571c-48e2-88b3-84a03dfbd80a" (UID: "decb2697-571c-48e2-88b3-84a03dfbd80a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:01:40.502252 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.502189 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-csr-dns-cert" pod "decb2697-571c-48e2-88b3-84a03dfbd80a" (UID: "decb2697-571c-48e2-88b3-84a03dfbd80a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:01:40.503069 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.503047 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/decb2697-571c-48e2-88b3-84a03dfbd80a-local-certs" pod "decb2697-571c-48e2-88b3-84a03dfbd80a" (UID: "decb2697-571c-48e2-88b3-84a03dfbd80a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:01:40.503362 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.503344 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-token" pod "decb2697-571c-48e2-88b3-84a03dfbd80a" (UID: "decb2697-571c-48e2-88b3-84a03dfbd80a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:01:40.504665 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.504642 5490 scope.go:122] "RemoveContainer" containerID="be2efe71c5d89dd064bd84b0cbaeb4f452545b619be5ec4cbc00703b6e0dc887" Oct 31 04:01:40.505001 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:01:40.504981 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be2efe71c5d89dd064bd84b0cbaeb4f452545b619be5ec4cbc00703b6e0dc887\": container with ID starting with be2efe71c5d89dd064bd84b0cbaeb4f452545b619be5ec4cbc00703b6e0dc887 not found: ID does not exist" containerID="be2efe71c5d89dd064bd84b0cbaeb4f452545b619be5ec4cbc00703b6e0dc887" Oct 31 04:01:40.505078 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.505011 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be2efe71c5d89dd064bd84b0cbaeb4f452545b619be5ec4cbc00703b6e0dc887"} err="failed to get container status \"be2efe71c5d89dd064bd84b0cbaeb4f452545b619be5ec4cbc00703b6e0dc887\": rpc error: code = NotFound desc = could not find container \"be2efe71c5d89dd064bd84b0cbaeb4f452545b619be5ec4cbc00703b6e0dc887\": container with ID starting with be2efe71c5d89dd064bd84b0cbaeb4f452545b619be5ec4cbc00703b6e0dc887 not found: ID does not exist" Oct 31 04:01:40.601391 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.601356 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:01:40.601391 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.601387 5490 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-kubeconfig\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:01:40.601391 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.601401 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g8sv\" (UniqueName: \"kubernetes.io/projected/decb2697-571c-48e2-88b3-84a03dfbd80a-kube-api-access-2g8sv\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:01:40.601645 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.601410 5490 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/decb2697-571c-48e2-88b3-84a03dfbd80a-local-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:01:40.601645 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.601420 5490 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-cacerts\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:01:40.601645 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.601428 5490 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/decb2697-571c-48e2-88b3-84a03dfbd80a-istio-csr-dns-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:01:40.816469 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.816423 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m"] Oct 31 04:01:40.820694 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:40.820649 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7x7m"] Oct 31 04:01:41.574911 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:41.574873 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="decb2697-571c-48e2-88b3-84a03dfbd80a" path="/var/lib/kubelet/pods/decb2697-571c-48e2-88b3-84a03dfbd80a/volumes" Oct 31 04:01:47.127269 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:01:47.127225 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:01:56.121267 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:01:56.121228 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:02:17.126977 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:02:17.126932 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:02:18.646755 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:02:18.646705 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:02:47.128240 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:02:47.128150 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:03:01.885290 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:03:01.885246 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:03:17.130456 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:03:17.130344 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:03:18.347399 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:03:18.347371 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ttmkb_400b676a-a3a0-459d-9da6-8d30b2d3b497/csi-driver/0.log" Oct 31 04:03:18.694222 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:03:18.694176 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:03:30.143944 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:03:30.143901 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:03:30.274915 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:03:30.274872 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:03:35.589204 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:03:35.589169 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ttmkb_400b676a-a3a0-459d-9da6-8d30b2d3b497/csi-driver/0.log" Oct 31 04:03:47.128356 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:03:47.128311 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:03:51.599801 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:03:51.597963 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-t9bq7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:03:51.599801 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:03:51.598479 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-t9bq7" podUID="6ffcf283-0d79-422e-82be-8b34db814efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:04:17.128424 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:04:17.128328 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:04:18.736198 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:04:18.736151 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:04:47.128296 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:04:47.128251 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:05:17.128103 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:05:17.128057 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:05:18.788564 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:18.788524 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:05:47.128241 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:05:47.128146 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:05:58.826740 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:58.826696 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:05:59.159022 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.158989 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g"] Oct 31 04:05:59.159508 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.159489 5490 cpu_manager.go:414] "containerMap: removing container" podUID="decb2697-571c-48e2-88b3-84a03dfbd80a" containerName="discovery" Oct 31 04:05:59.159611 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.159512 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="decb2697-571c-48e2-88b3-84a03dfbd80a" containerName="discovery" Oct 31 04:05:59.159715 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.159702 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="decb2697-571c-48e2-88b3-84a03dfbd80a" containerName="discovery" Oct 31 04:05:59.162739 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.162719 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.170554 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.170459 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-799962-openshift-default-dockercfg-vmhxw\"" Oct 31 04:05:59.174404 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.174379 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g"] Oct 31 04:05:59.265994 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.265952 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-credential-socket\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.266184 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.266000 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-token\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.266184 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.266027 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-workload-socket\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.266184 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.266150 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-workload-certs\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.266333 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.266187 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfrht\" (UniqueName: \"kubernetes.io/projected/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-kube-api-access-wfrht\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.266333 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.266242 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-envoy\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.266333 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.266258 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-data\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.266333 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.266281 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-podinfo\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.266479 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.266368 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istiod-ca-cert\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.367168 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.367131 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-credential-socket\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.367372 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.367178 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-token\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.367372 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.367199 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-workload-socket\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.367372 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.367245 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-workload-certs\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.367372 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.367268 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfrht\" (UniqueName: \"kubernetes.io/projected/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-kube-api-access-wfrht\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.367372 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.367300 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-envoy\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.367372 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.367315 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-data\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.367372 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.367346 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-podinfo\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.367825 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.367418 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istiod-ca-cert\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.367825 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.367645 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-credential-socket\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.367825 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.367702 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-workload-socket\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.367825 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.367778 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-workload-certs\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.368059 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.368032 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-data\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.368233 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.368215 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istiod-ca-cert\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.369783 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.369757 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-envoy\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.369905 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.369879 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-podinfo\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.374765 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.374745 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-token\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.375235 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.375218 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfrht\" (UniqueName: \"kubernetes.io/projected/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-kube-api-access-wfrht\") pod \"e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.479276 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.479184 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:05:59.763502 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.763466 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g"] Oct 31 04:05:59.764713 ip-10-0-142-76 kubenswrapper[5490]: W1031 04:05:59.764660 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86282a86_4bfb_4093_b5eb_402c8b1ea4a2.slice/crio-f6cbb4dba442f4571c32c53c8f80dfac84d72bb26f12a8ba309127bc5e92ee04 WatchSource:0}: Error finding container f6cbb4dba442f4571c32c53c8f80dfac84d72bb26f12a8ba309127bc5e92ee04: Status 404 returned error can't find the container with id f6cbb4dba442f4571c32c53c8f80dfac84d72bb26f12a8ba309127bc5e92ee04 Oct 31 04:05:59.767055 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.767019 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:05:59.767165 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.767103 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:05:59.767227 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:05:59.767173 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:06:00.228937 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:00.228895 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" event={"ID":"86282a86-4bfb-4093-b5eb-402c8b1ea4a2","Type":"ContainerStarted","Data":"e11c1503284b00bd98daa4b2a8326ac22d127a53bc2cb19eb5fb70abb294db91"} Oct 31 04:06:00.229464 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:00.228966 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" event={"ID":"86282a86-4bfb-4093-b5eb-402c8b1ea4a2","Type":"ContainerStarted","Data":"f6cbb4dba442f4571c32c53c8f80dfac84d72bb26f12a8ba309127bc5e92ee04"} Oct 31 04:06:00.230020 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:00.229992 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:00.249692 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:00.249617 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" podStartSLOduration=1.24959912 podStartE2EDuration="1.24959912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:06:00.248244984 +0000 UTC m=+2994.782939939" watchObservedRunningTime="2026-10-31 04:06:00.24959912 +0000 UTC m=+2994.784294075" Oct 31 04:06:00.479760 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:00.479635 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:06:00.485098 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:00.485067 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:06:01.234616 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:01.234587 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:06:01.236970 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:01.235703 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:06:02.435498 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:02.435451 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:02.611042 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:02.611005 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:03.603876 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:03.603841 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:03.692388 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:03.692355 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw"] Oct 31 04:06:03.696160 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:03.696140 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw" Oct 31 04:06:03.703658 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:03.703630 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-799962\"" Oct 31 04:06:03.703934 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:03.703748 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-0bd916eb-dockercfg-kkpcl\"" Oct 31 04:06:03.707283 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:03.707261 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw"] Oct 31 04:06:03.752361 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:03.752327 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:03.813610 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:03.813564 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eaab956-e533-4b8d-9042-f5858e16dbc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw\" (UID: \"3eaab956-e533-4b8d-9042-f5858e16dbc0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw" Oct 31 04:06:03.813788 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:03.813617 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sws6n\" (UniqueName: \"kubernetes.io/projected/3eaab956-e533-4b8d-9042-f5858e16dbc0-kube-api-access-sws6n\") pod \"payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw\" (UID: \"3eaab956-e533-4b8d-9042-f5858e16dbc0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw" Oct 31 04:06:03.914782 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:03.914744 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eaab956-e533-4b8d-9042-f5858e16dbc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw\" (UID: \"3eaab956-e533-4b8d-9042-f5858e16dbc0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw" Oct 31 04:06:03.914980 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:03.914798 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sws6n\" (UniqueName: \"kubernetes.io/projected/3eaab956-e533-4b8d-9042-f5858e16dbc0-kube-api-access-sws6n\") pod \"payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw\" (UID: \"3eaab956-e533-4b8d-9042-f5858e16dbc0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw" Oct 31 04:06:03.915539 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:03.915518 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eaab956-e533-4b8d-9042-f5858e16dbc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw\" (UID: \"3eaab956-e533-4b8d-9042-f5858e16dbc0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw" Oct 31 04:06:03.923031 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:03.923002 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sws6n\" (UniqueName: \"kubernetes.io/projected/3eaab956-e533-4b8d-9042-f5858e16dbc0-kube-api-access-sws6n\") pod \"payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw\" (UID: \"3eaab956-e533-4b8d-9042-f5858e16dbc0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw" Oct 31 04:06:04.011910 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:04.011865 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw" Oct 31 04:06:04.300225 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:04.300194 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw"] Oct 31 04:06:04.302374 ip-10-0-142-76 kubenswrapper[5490]: W1031 04:06:04.302345 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eaab956_e533_4b8d_9042_f5858e16dbc0.slice/crio-f10b366d7273558f162fd33222208a5fc3906e8689575a32809e2be8db5b14c5 WatchSource:0}: Error finding container f10b366d7273558f162fd33222208a5fc3906e8689575a32809e2be8db5b14c5: Status 404 returned error can't find the container with id f10b366d7273558f162fd33222208a5fc3906e8689575a32809e2be8db5b14c5 Oct 31 04:06:04.377662 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:04.377632 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:05.092126 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:05.092073 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:05.261505 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:05.261446 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw" event={"ID":"3eaab956-e533-4b8d-9042-f5858e16dbc0","Type":"ContainerStarted","Data":"b89e51585b9ea743b41f6aeca65d5e7c8fb471a61249863fdde17c94ee3b805f"} Oct 31 04:06:05.261723 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:05.261558 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw" event={"ID":"3eaab956-e533-4b8d-9042-f5858e16dbc0","Type":"ContainerStarted","Data":"f10b366d7273558f162fd33222208a5fc3906e8689575a32809e2be8db5b14c5"} Oct 31 04:06:05.261723 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:05.261592 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw" Oct 31 04:06:05.280386 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:05.280330 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw" podStartSLOduration=2.280315108 podStartE2EDuration="2.280315108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:06:05.277166064 +0000 UTC m=+2999.811861011" watchObservedRunningTime="2026-10-31 04:06:05.280315108 +0000 UTC m=+2999.815010063" Oct 31 04:06:05.699266 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:05.699221 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:06.559963 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:06.559920 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:07.225182 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:07.225142 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:07.919799 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:07.919734 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:08.819165 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:08.819130 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:09.603328 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:09.603291 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:10.321842 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:10.321793 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:11.099957 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:11.099913 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:11.953705 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:11.953638 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:12.343935 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:12.343902 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:12.599236 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:12.599104 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:12.878027 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:12.877916 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:13.026013 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:13.025978 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:13.164015 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:13.163975 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:13.410576 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:13.410535 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:14.164855 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:14.164818 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:14.636065 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:14.635968 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:14.817713 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:14.817636 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:15.321634 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:15.321539 5490 generic.go:356] "Generic (PLEG): container finished" podID="86282a86-4bfb-4093-b5eb-402c8b1ea4a2" containerID="e11c1503284b00bd98daa4b2a8326ac22d127a53bc2cb19eb5fb70abb294db91" exitCode=0 Oct 31 04:06:15.321634 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:15.321612 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" event={"ID":"86282a86-4bfb-4093-b5eb-402c8b1ea4a2","Type":"ContainerDied","Data":"e11c1503284b00bd98daa4b2a8326ac22d127a53bc2cb19eb5fb70abb294db91"} Oct 31 04:06:15.322080 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:15.322063 5490 scope.go:122] "RemoveContainer" containerID="e11c1503284b00bd98daa4b2a8326ac22d127a53bc2cb19eb5fb70abb294db91" Oct 31 04:06:15.323440 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:15.323404 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:06:15.323699 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:15.323643 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:06:15.323806 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:15.323720 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:06:15.424358 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:15.424315 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:16.106783 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:16.106734 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:16.240709 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:16.239790 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:06:16.271403 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:16.271367 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw" Oct 31 04:06:16.361759 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:16.360380 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" event={"ID":"86282a86-4bfb-4093-b5eb-402c8b1ea4a2","Type":"ContainerStarted","Data":"075864d769837079faea195eb771491f53e5b89660af2b61120ed36d977f5b9e"} Oct 31 04:06:16.480466 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:16.480116 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:06:16.481399 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:16.481368 5490 patch_prober.go:29] interesting pod/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 04:06:16.481561 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:16.481442 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" podUID="86282a86-4bfb-4093-b5eb-402c8b1ea4a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 04:06:16.913448 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:16.913410 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:17.152625 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:06:17.152547 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:06:17.479727 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:17.479669 5490 patch_prober.go:29] interesting pod/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 04:06:17.479942 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:17.479740 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" podUID="86282a86-4bfb-4093-b5eb-402c8b1ea4a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 04:06:17.610634 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:17.610592 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:18.310075 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:18.310040 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:18.480091 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:18.480050 5490 patch_prober.go:29] interesting pod/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 04:06:18.480278 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:18.480129 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" podUID="86282a86-4bfb-4093-b5eb-402c8b1ea4a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 04:06:18.831788 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:18.831749 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:19.077887 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:19.077853 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:19.479806 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:19.479769 5490 patch_prober.go:29] interesting pod/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 04:06:19.480210 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:19.479841 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" podUID="86282a86-4bfb-4093-b5eb-402c8b1ea4a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 04:06:19.807134 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:19.807031 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:20.480646 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:20.480598 5490 patch_prober.go:29] interesting pod/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 04:06:20.481160 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:20.480707 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" podUID="86282a86-4bfb-4093-b5eb-402c8b1ea4a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 04:06:20.717127 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:20.717088 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:21.406770 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:21.406731 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:21.480607 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:21.480565 5490 patch_prober.go:29] interesting pod/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 04:06:21.480860 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:21.480638 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" podUID="86282a86-4bfb-4093-b5eb-402c8b1ea4a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 04:06:22.088269 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:22.088231 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:22.484349 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:22.484309 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:06:22.484838 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:22.484719 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:06:22.485777 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:22.485758 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:06:22.707013 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:22.706980 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:23.380158 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:23.380117 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:23.991110 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:23.991074 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:24.693384 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:24.693343 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:25.289596 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:25.289542 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:25.897857 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:25.897805 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:26.508776 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:26.508739 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:27.212369 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:27.212325 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:27.806698 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:27.806625 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:28.486206 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:28.486168 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:29.084143 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:29.084110 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:29.816887 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:29.816850 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:30.504774 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:30.504736 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:31.178992 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:31.178950 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:31.997257 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:31.997213 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:32.790729 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:32.790694 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:33.496239 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:33.496201 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:34.197960 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:34.197866 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:34.797303 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:34.797254 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:35.407356 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:35.407314 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:36.006730 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:36.006602 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:36.602612 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:36.602574 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:37.283396 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:37.283360 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:37.998165 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:37.998114 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:38.595279 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:38.595158 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:39.205422 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:39.205387 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:39.810871 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:39.810830 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:40.492503 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:40.492462 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:41.100910 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:41.100866 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:41.760311 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:41.760270 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:42.319812 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:42.319768 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:42.982421 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:42.982373 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:43.588900 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:43.588852 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:44.397512 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:44.397464 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:45.074483 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:45.074435 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:45.717934 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:45.717890 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:46.474950 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:46.474910 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:47.129220 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:06:47.129183 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:06:47.221580 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:47.221534 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:47.887651 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:47.887611 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:48.576990 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:48.576957 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:49.168808 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:49.168768 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:49.852290 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:49.852249 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:50.053622 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:50.053584 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:50.618365 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:50.618319 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:51.311610 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:51.311573 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:52.006222 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:52.006169 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:52.604404 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:52.604367 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:53.300310 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:53.300271 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:53.899773 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:53.899733 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:54.607875 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:54.607810 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:55.044303 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:55.044261 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:55.206006 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:55.205970 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:55.995477 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:55.995434 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:56.715266 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:56.715225 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:57.587149 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:57.587110 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:58.314406 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:58.314368 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:59.155736 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:59.155696 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:06:59.711590 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:06:59.711546 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:00.068714 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:00.068590 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:00.377971 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:00.377872 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:01.049821 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:01.049770 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:01.698464 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:01.698424 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:02.294722 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:02.294654 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:02.905955 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:02.905908 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:03.504545 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:03.504504 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:04.106305 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:04.106262 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:04.703000 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:04.702962 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:05.254936 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:05.254901 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw"] Oct 31 04:07:05.300231 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:05.300173 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:05.431345 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:05.431306 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:05.894546 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:05.894496 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:06.421598 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:06.421556 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:06.550740 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:06.550703 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:06.790482 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:06.790386 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:07.080437 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:07.080339 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:10.255911 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:10.255865 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw" podUID="3eaab956-e533-4b8d-9042-f5858e16dbc0" containerName="payload-pre-processing" containerID="cri-o://b89e51585b9ea743b41f6aeca65d5e7c8fb471a61249863fdde17c94ee3b805f" gracePeriod=25 Oct 31 04:07:12.905421 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:12.905383 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:13.037876 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:13.037831 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:13.155261 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:13.155223 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g"] Oct 31 04:07:13.155547 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:13.155485 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" podUID="86282a86-4bfb-4093-b5eb-402c8b1ea4a2" containerName="istio-proxy" containerID="cri-o://075864d769837079faea195eb771491f53e5b89660af2b61120ed36d977f5b9e" gracePeriod=30 Oct 31 04:07:13.243911 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:13.243858 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:16.269483 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:16.269392 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw" podUID="3eaab956-e533-4b8d-9042-f5858e16dbc0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:07:17.127496 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:07:17.127457 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:07:18.409582 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.409556 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:07:18.511064 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.510968 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-workload-socket\") pod \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " Oct 31 04:07:18.511064 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.511036 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-token\" (UniqueName: \"kubernetes.io/projected/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-token\") pod \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " Oct 31 04:07:18.511309 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.511069 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istiod-ca-cert\") pod \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " Oct 31 04:07:18.511309 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.511136 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-envoy\") pod \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " Oct 31 04:07:18.511309 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.511160 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-workload-certs\") pod \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " Oct 31 04:07:18.511309 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.511236 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-workload-socket" pod "86282a86-4bfb-4093-b5eb-402c8b1ea4a2" (UID: "86282a86-4bfb-4093-b5eb-402c8b1ea4a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:18.511309 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.511252 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-podinfo\") pod \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " Oct 31 04:07:18.511309 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.511296 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-kube-api-access-wfrht\" (UniqueName: \"kubernetes.io/projected/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-kube-api-access-wfrht\") pod \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " Oct 31 04:07:18.511659 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.511320 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-data\") pod \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " Oct 31 04:07:18.511659 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.511341 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-credential-socket\") pod \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\" (UID: \"86282a86-4bfb-4093-b5eb-402c8b1ea4a2\") " Oct 31 04:07:18.511659 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.511616 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-data" pod "86282a86-4bfb-4093-b5eb-402c8b1ea4a2" (UID: "86282a86-4bfb-4093-b5eb-402c8b1ea4a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:18.511659 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.511644 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istiod-ca-cert" pod "86282a86-4bfb-4093-b5eb-402c8b1ea4a2" (UID: "86282a86-4bfb-4093-b5eb-402c8b1ea4a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:07:18.511913 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.511773 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-workload-certs" pod "86282a86-4bfb-4093-b5eb-402c8b1ea4a2" (UID: "86282a86-4bfb-4093-b5eb-402c8b1ea4a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:18.511913 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.511790 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:18.511913 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.511846 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:18.511913 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.511852 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-credential-socket" pod "86282a86-4bfb-4093-b5eb-402c8b1ea4a2" (UID: "86282a86-4bfb-4093-b5eb-402c8b1ea4a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:18.513463 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.513429 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-envoy" pod "86282a86-4bfb-4093-b5eb-402c8b1ea4a2" (UID: "86282a86-4bfb-4093-b5eb-402c8b1ea4a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:07:18.513574 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.513545 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-kube-api-access-wfrht" pod "86282a86-4bfb-4093-b5eb-402c8b1ea4a2" (UID: "86282a86-4bfb-4093-b5eb-402c8b1ea4a2"). InnerVolumeSpecName "kube-api-access-wfrht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:18.513785 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.513752 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-podinfo" pod "86282a86-4bfb-4093-b5eb-402c8b1ea4a2" (UID: "86282a86-4bfb-4093-b5eb-402c8b1ea4a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:07:18.513785 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.513770 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-token" pod "86282a86-4bfb-4093-b5eb-402c8b1ea4a2" (UID: "86282a86-4bfb-4093-b5eb-402c8b1ea4a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:18.612885 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.612847 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:18.612885 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.612883 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:18.613129 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.612899 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:18.613129 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.612912 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:18.613129 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.612923 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:18.613129 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.612932 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfrht\" (UniqueName: \"kubernetes.io/projected/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-kube-api-access-wfrht\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:18.613129 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.612942 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86282a86-4bfb-4093-b5eb-402c8b1ea4a2-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:18.753869 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.753832 5490 generic.go:356] "Generic (PLEG): container finished" podID="86282a86-4bfb-4093-b5eb-402c8b1ea4a2" containerID="075864d769837079faea195eb771491f53e5b89660af2b61120ed36d977f5b9e" exitCode=0 Oct 31 04:07:18.754054 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.753940 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" Oct 31 04:07:18.754054 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.753947 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" event={"ID":"86282a86-4bfb-4093-b5eb-402c8b1ea4a2","Type":"ContainerDied","Data":"075864d769837079faea195eb771491f53e5b89660af2b61120ed36d977f5b9e"} Oct 31 04:07:18.754135 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.754054 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g" event={"ID":"86282a86-4bfb-4093-b5eb-402c8b1ea4a2","Type":"ContainerDied","Data":"f6cbb4dba442f4571c32c53c8f80dfac84d72bb26f12a8ba309127bc5e92ee04"} Oct 31 04:07:18.754135 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.754076 5490 scope.go:122] "RemoveContainer" containerID="075864d769837079faea195eb771491f53e5b89660af2b61120ed36d977f5b9e" Oct 31 04:07:18.764669 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.764648 5490 scope.go:122] "RemoveContainer" containerID="e11c1503284b00bd98daa4b2a8326ac22d127a53bc2cb19eb5fb70abb294db91" Oct 31 04:07:18.778562 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.778527 5490 scope.go:122] "RemoveContainer" containerID="075864d769837079faea195eb771491f53e5b89660af2b61120ed36d977f5b9e" Oct 31 04:07:18.778990 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:07:18.778965 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"075864d769837079faea195eb771491f53e5b89660af2b61120ed36d977f5b9e\": container with ID starting with 075864d769837079faea195eb771491f53e5b89660af2b61120ed36d977f5b9e not found: ID does not exist" containerID="075864d769837079faea195eb771491f53e5b89660af2b61120ed36d977f5b9e" Oct 31 04:07:18.779061 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.779003 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"075864d769837079faea195eb771491f53e5b89660af2b61120ed36d977f5b9e"} err="failed to get container status \"075864d769837079faea195eb771491f53e5b89660af2b61120ed36d977f5b9e\": rpc error: code = NotFound desc = could not find container \"075864d769837079faea195eb771491f53e5b89660af2b61120ed36d977f5b9e\": container with ID starting with 075864d769837079faea195eb771491f53e5b89660af2b61120ed36d977f5b9e not found: ID does not exist" Oct 31 04:07:18.779061 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.779024 5490 scope.go:122] "RemoveContainer" containerID="e11c1503284b00bd98daa4b2a8326ac22d127a53bc2cb19eb5fb70abb294db91" Oct 31 04:07:18.779334 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:07:18.779314 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e11c1503284b00bd98daa4b2a8326ac22d127a53bc2cb19eb5fb70abb294db91\": container with ID starting with e11c1503284b00bd98daa4b2a8326ac22d127a53bc2cb19eb5fb70abb294db91 not found: ID does not exist" containerID="e11c1503284b00bd98daa4b2a8326ac22d127a53bc2cb19eb5fb70abb294db91" Oct 31 04:07:18.779379 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.779341 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e11c1503284b00bd98daa4b2a8326ac22d127a53bc2cb19eb5fb70abb294db91"} err="failed to get container status \"e11c1503284b00bd98daa4b2a8326ac22d127a53bc2cb19eb5fb70abb294db91\": rpc error: code = NotFound desc = could not find container \"e11c1503284b00bd98daa4b2a8326ac22d127a53bc2cb19eb5fb70abb294db91\": container with ID starting with e11c1503284b00bd98daa4b2a8326ac22d127a53bc2cb19eb5fb70abb294db91 not found: ID does not exist" Oct 31 04:07:18.779544 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.779524 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g"] Oct 31 04:07:18.785220 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.785194 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-799962-openshift-default-6d9b4c97d96kj9g"] Oct 31 04:07:18.878074 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:18.878039 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:19.573842 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:19.573799 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86282a86-4bfb-4093-b5eb-402c8b1ea4a2" path="/var/lib/kubelet/pods/86282a86-4bfb-4093-b5eb-402c8b1ea4a2/volumes" Oct 31 04:07:26.293336 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:26.293281 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw" podUID="3eaab956-e533-4b8d-9042-f5858e16dbc0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:07:30.512190 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:30.512120 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw" Oct 31 04:07:30.543951 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:30.543919 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eaab956-e533-4b8d-9042-f5858e16dbc0-kube-api-access-sws6n\" (UniqueName: \"kubernetes.io/projected/3eaab956-e533-4b8d-9042-f5858e16dbc0-kube-api-access-sws6n\") pod \"3eaab956-e533-4b8d-9042-f5858e16dbc0\" (UID: \"3eaab956-e533-4b8d-9042-f5858e16dbc0\") " Oct 31 04:07:30.544119 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:30.543982 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eaab956-e533-4b8d-9042-f5858e16dbc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eaab956-e533-4b8d-9042-f5858e16dbc0-plugins-config-volume\") pod \"3eaab956-e533-4b8d-9042-f5858e16dbc0\" (UID: \"3eaab956-e533-4b8d-9042-f5858e16dbc0\") " Oct 31 04:07:30.544433 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:30.544400 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eaab956-e533-4b8d-9042-f5858e16dbc0-plugins-config-volume" pod "3eaab956-e533-4b8d-9042-f5858e16dbc0" (UID: "3eaab956-e533-4b8d-9042-f5858e16dbc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:07:30.546244 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:30.546217 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eaab956-e533-4b8d-9042-f5858e16dbc0-kube-api-access-sws6n" pod "3eaab956-e533-4b8d-9042-f5858e16dbc0" (UID: "3eaab956-e533-4b8d-9042-f5858e16dbc0"). InnerVolumeSpecName "kube-api-access-sws6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:07:30.646390 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:30.646313 5490 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3eaab956-e533-4b8d-9042-f5858e16dbc0-plugins-config-volume\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:30.646390 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:30.646347 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sws6n\" (UniqueName: \"kubernetes.io/projected/3eaab956-e533-4b8d-9042-f5858e16dbc0-kube-api-access-sws6n\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:07:30.833208 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:30.833121 5490 generic.go:356] "Generic (PLEG): container finished" podID="3eaab956-e533-4b8d-9042-f5858e16dbc0" containerID="b89e51585b9ea743b41f6aeca65d5e7c8fb471a61249863fdde17c94ee3b805f" exitCode=0 Oct 31 04:07:30.833208 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:30.833171 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw" Oct 31 04:07:30.833404 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:30.833212 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw" event={"ID":"3eaab956-e533-4b8d-9042-f5858e16dbc0","Type":"ContainerDied","Data":"b89e51585b9ea743b41f6aeca65d5e7c8fb471a61249863fdde17c94ee3b805f"} Oct 31 04:07:30.833404 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:30.833277 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw" event={"ID":"3eaab956-e533-4b8d-9042-f5858e16dbc0","Type":"ContainerDied","Data":"f10b366d7273558f162fd33222208a5fc3906e8689575a32809e2be8db5b14c5"} Oct 31 04:07:30.833404 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:30.833305 5490 scope.go:122] "RemoveContainer" containerID="b89e51585b9ea743b41f6aeca65d5e7c8fb471a61249863fdde17c94ee3b805f" Oct 31 04:07:30.842744 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:30.842710 5490 scope.go:122] "RemoveContainer" containerID="b89e51585b9ea743b41f6aeca65d5e7c8fb471a61249863fdde17c94ee3b805f" Oct 31 04:07:30.843073 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:07:30.843043 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89e51585b9ea743b41f6aeca65d5e7c8fb471a61249863fdde17c94ee3b805f\": container with ID starting with b89e51585b9ea743b41f6aeca65d5e7c8fb471a61249863fdde17c94ee3b805f not found: ID does not exist" containerID="b89e51585b9ea743b41f6aeca65d5e7c8fb471a61249863fdde17c94ee3b805f" Oct 31 04:07:30.843173 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:30.843085 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89e51585b9ea743b41f6aeca65d5e7c8fb471a61249863fdde17c94ee3b805f"} err="failed to get container status \"b89e51585b9ea743b41f6aeca65d5e7c8fb471a61249863fdde17c94ee3b805f\": rpc error: code = NotFound desc = could not find container \"b89e51585b9ea743b41f6aeca65d5e7c8fb471a61249863fdde17c94ee3b805f\": container with ID starting with b89e51585b9ea743b41f6aeca65d5e7c8fb471a61249863fdde17c94ee3b805f not found: ID does not exist" Oct 31 04:07:30.855010 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:30.854980 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw"] Oct 31 04:07:30.861360 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:30.861333 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-799962-85b565jjrw"] Oct 31 04:07:31.574230 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:31.574193 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eaab956-e533-4b8d-9042-f5858e16dbc0" path="/var/lib/kubelet/pods/3eaab956-e533-4b8d-9042-f5858e16dbc0/volumes" Oct 31 04:07:36.507137 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:36.507077 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:37.621284 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:37.621246 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:37.738722 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:37.738662 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:43.359264 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:43.359228 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:47.126912 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:07:47.126865 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:07:55.217884 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:55.217844 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:59.644732 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:59.644689 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:07:59.765752 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:07:59.765710 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:08:17.071135 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:08:17.071078 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:08:17.128218 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:08:17.128179 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:08:18.385780 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:08:18.385754 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ttmkb_400b676a-a3a0-459d-9da6-8d30b2d3b497/csi-driver/0.log" Oct 31 04:08:18.956087 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:08:18.956041 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:08:35.641575 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:08:35.641547 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ttmkb_400b676a-a3a0-459d-9da6-8d30b2d3b497/csi-driver/0.log" Oct 31 04:08:47.128518 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:08:47.128432 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:09:17.127584 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:09:17.127542 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:09:20.672689 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:20.672631 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:26.634402 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:26.634360 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:26.934823 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:26.934786 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp"] Oct 31 04:09:26.935364 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:26.935344 5490 cpu_manager.go:414] "containerMap: removing container" podUID="3eaab956-e533-4b8d-9042-f5858e16dbc0" containerName="payload-pre-processing" Oct 31 04:09:26.935415 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:26.935367 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eaab956-e533-4b8d-9042-f5858e16dbc0" containerName="payload-pre-processing" Oct 31 04:09:26.935415 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:26.935388 5490 cpu_manager.go:414] "containerMap: removing container" podUID="86282a86-4bfb-4093-b5eb-402c8b1ea4a2" containerName="istio-proxy" Oct 31 04:09:26.935415 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:26.935393 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86282a86-4bfb-4093-b5eb-402c8b1ea4a2" containerName="istio-proxy" Oct 31 04:09:26.935521 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:26.935493 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="86282a86-4bfb-4093-b5eb-402c8b1ea4a2" containerName="istio-proxy" Oct 31 04:09:26.935521 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:26.935501 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eaab956-e533-4b8d-9042-f5858e16dbc0" containerName="payload-pre-processing" Oct 31 04:09:26.935521 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:26.935509 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="86282a86-4bfb-4093-b5eb-402c8b1ea4a2" containerName="istio-proxy" Oct 31 04:09:26.938847 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:26.938824 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:26.947784 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:26.947754 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-8a336e-openshift-default-dockercfg-zggjd\"" Oct 31 04:09:26.951656 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:26.951626 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp"] Oct 31 04:09:27.024546 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.024504 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:27.038989 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.038954 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-token\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.039213 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.039001 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-credential-socket\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.039213 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.039037 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-workload-certs\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.039213 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.039070 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh9tv\" (UniqueName: \"kubernetes.io/projected/f28ec478-90c6-4623-8e9a-6d43b9311629-kube-api-access-xh9tv\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.039213 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.039092 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-workload-socket\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.039213 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.039110 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-data\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.039213 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.039139 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28ec478-90c6-4623-8e9a-6d43b9311629-istiod-ca-cert\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.039213 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.039166 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-podinfo\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.039213 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.039187 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-envoy\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.139761 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.139718 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28ec478-90c6-4623-8e9a-6d43b9311629-istiod-ca-cert\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.139989 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.139784 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-podinfo\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.139989 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.139821 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-envoy\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.139989 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.139856 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-token\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.139989 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.139898 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-credential-socket\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.139989 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.139952 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-workload-certs\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.140315 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.140013 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh9tv\" (UniqueName: \"kubernetes.io/projected/f28ec478-90c6-4623-8e9a-6d43b9311629-kube-api-access-xh9tv\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.140315 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.140052 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-workload-socket\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.140315 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.140079 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-data\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.140480 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.140360 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-credential-socket\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.140480 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.140440 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-data\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.140586 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.140495 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-workload-certs\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.140586 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.140532 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28ec478-90c6-4623-8e9a-6d43b9311629-istiod-ca-cert\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.140586 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.140558 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-workload-socket\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.142241 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.142219 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-envoy\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.142364 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.142311 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-podinfo\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.150707 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.150651 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-token\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.150955 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.150929 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh9tv\" (UniqueName: \"kubernetes.io/projected/f28ec478-90c6-4623-8e9a-6d43b9311629-kube-api-access-xh9tv\") pod \"e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.171443 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.171405 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:27.254607 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.254513 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:27.535468 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.535441 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp"] Oct 31 04:09:27.537139 ip-10-0-142-76 kubenswrapper[5490]: W1031 04:09:27.537101 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28ec478_90c6_4623_8e9a_6d43b9311629.slice/crio-761dcf276cf6ff1a155d165375f3e2babc0e8229f8833cd99cc9058a1ac7713a WatchSource:0}: Error finding container 761dcf276cf6ff1a155d165375f3e2babc0e8229f8833cd99cc9058a1ac7713a: Status 404 returned error can't find the container with id 761dcf276cf6ff1a155d165375f3e2babc0e8229f8833cd99cc9058a1ac7713a Oct 31 04:09:27.539402 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.539365 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:09:27.539497 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.539437 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:09:27.539497 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:27.539469 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:09:28.506900 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:28.506858 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" event={"ID":"f28ec478-90c6-4623-8e9a-6d43b9311629","Type":"ContainerStarted","Data":"68e7d903940736da4506d425d748a892c87b768e5e46e89977e115dacd2432bf"} Oct 31 04:09:28.507338 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:28.506919 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" event={"ID":"f28ec478-90c6-4623-8e9a-6d43b9311629","Type":"ContainerStarted","Data":"761dcf276cf6ff1a155d165375f3e2babc0e8229f8833cd99cc9058a1ac7713a"} Oct 31 04:09:28.529254 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:28.529186 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" podStartSLOduration=2.529166369 podStartE2EDuration="2.529166369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:28.52723987 +0000 UTC m=+3203.061934824" watchObservedRunningTime="2026-10-31 04:09:28.529166369 +0000 UTC m=+3203.063861325" Oct 31 04:09:29.255692 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:29.255645 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:29.261131 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:29.261103 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:29.512823 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:29.512739 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:29.513782 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:29.513764 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:30.764907 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:30.764871 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:30.899909 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:30.899874 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:31.926078 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:31.926040 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5"] Oct 31 04:09:31.926783 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:31.926759 5490 cpu_manager.go:414] "containerMap: removing container" podUID="86282a86-4bfb-4093-b5eb-402c8b1ea4a2" containerName="istio-proxy" Oct 31 04:09:31.926878 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:31.926785 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86282a86-4bfb-4093-b5eb-402c8b1ea4a2" containerName="istio-proxy" Oct 31 04:09:31.943121 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:31.943085 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5"] Oct 31 04:09:31.943380 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:31.943351 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" Oct 31 04:09:31.947876 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:31.947850 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-8a336e-serving-cert\"" Oct 31 04:09:31.978198 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:31.978155 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:31.986579 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:31.986540 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06c20927-b1f8-4930-8a01-5a314fce9efc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5\" (UID: \"06c20927-b1f8-4930-8a01-5a314fce9efc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" Oct 31 04:09:31.986828 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:31.986591 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2l9s\" (UniqueName: \"kubernetes.io/projected/06c20927-b1f8-4930-8a01-5a314fce9efc-kube-api-access-x2l9s\") pod \"maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5\" (UID: \"06c20927-b1f8-4930-8a01-5a314fce9efc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" Oct 31 04:09:31.986828 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:31.986740 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06c20927-b1f8-4930-8a01-5a314fce9efc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5\" (UID: \"06c20927-b1f8-4930-8a01-5a314fce9efc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" Oct 31 04:09:31.986828 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:31.986793 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c20927-b1f8-4930-8a01-5a314fce9efc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5\" (UID: \"06c20927-b1f8-4930-8a01-5a314fce9efc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" Oct 31 04:09:32.052643 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.052602 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm"] Oct 31 04:09:32.056623 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.056597 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm" Oct 31 04:09:32.065222 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.065194 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-8abb7425-dockercfg-ct2xs\"" Oct 31 04:09:32.065605 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.065575 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm"] Oct 31 04:09:32.088386 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.088347 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06c20927-b1f8-4930-8a01-5a314fce9efc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5\" (UID: \"06c20927-b1f8-4930-8a01-5a314fce9efc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" Oct 31 04:09:32.088611 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.088415 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c20927-b1f8-4930-8a01-5a314fce9efc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5\" (UID: \"06c20927-b1f8-4930-8a01-5a314fce9efc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" Oct 31 04:09:32.088611 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.088464 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dbsf\" (UniqueName: \"kubernetes.io/projected/9eeef949-c2c4-48e8-8099-ab62fd54f865-kube-api-access-5dbsf\") pod \"payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm\" (UID: \"9eeef949-c2c4-48e8-8099-ab62fd54f865\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm" Oct 31 04:09:32.088611 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.088505 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06c20927-b1f8-4930-8a01-5a314fce9efc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5\" (UID: \"06c20927-b1f8-4930-8a01-5a314fce9efc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" Oct 31 04:09:32.088611 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.088599 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eeef949-c2c4-48e8-8099-ab62fd54f865-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm\" (UID: \"9eeef949-c2c4-48e8-8099-ab62fd54f865\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm" Oct 31 04:09:32.088943 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.088640 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2l9s\" (UniqueName: \"kubernetes.io/projected/06c20927-b1f8-4930-8a01-5a314fce9efc-kube-api-access-x2l9s\") pod \"maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5\" (UID: \"06c20927-b1f8-4930-8a01-5a314fce9efc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" Oct 31 04:09:32.089202 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.089177 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c20927-b1f8-4930-8a01-5a314fce9efc-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5\" (UID: \"06c20927-b1f8-4930-8a01-5a314fce9efc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" Oct 31 04:09:32.091128 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.091103 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06c20927-b1f8-4930-8a01-5a314fce9efc-metrics-certs\") pod \"maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5\" (UID: \"06c20927-b1f8-4930-8a01-5a314fce9efc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" Oct 31 04:09:32.091338 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.091314 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06c20927-b1f8-4930-8a01-5a314fce9efc-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5\" (UID: \"06c20927-b1f8-4930-8a01-5a314fce9efc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" Oct 31 04:09:32.099145 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.099117 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2l9s\" (UniqueName: \"kubernetes.io/projected/06c20927-b1f8-4930-8a01-5a314fce9efc-kube-api-access-x2l9s\") pod \"maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5\" (UID: \"06c20927-b1f8-4930-8a01-5a314fce9efc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" Oct 31 04:09:32.108965 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.108936 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-8a336e\"" Oct 31 04:09:32.123838 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.123795 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:32.190465 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.190326 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dbsf\" (UniqueName: \"kubernetes.io/projected/9eeef949-c2c4-48e8-8099-ab62fd54f865-kube-api-access-5dbsf\") pod \"payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm\" (UID: \"9eeef949-c2c4-48e8-8099-ab62fd54f865\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm" Oct 31 04:09:32.190822 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.190551 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eeef949-c2c4-48e8-8099-ab62fd54f865-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm\" (UID: \"9eeef949-c2c4-48e8-8099-ab62fd54f865\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm" Oct 31 04:09:32.191658 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.191629 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eeef949-c2c4-48e8-8099-ab62fd54f865-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm\" (UID: \"9eeef949-c2c4-48e8-8099-ab62fd54f865\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm" Oct 31 04:09:32.201559 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.201527 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dbsf\" (UniqueName: \"kubernetes.io/projected/9eeef949-c2c4-48e8-8099-ab62fd54f865-kube-api-access-5dbsf\") pod \"payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm\" (UID: \"9eeef949-c2c4-48e8-8099-ab62fd54f865\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm" Oct 31 04:09:32.253798 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.253762 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:32.260849 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.260815 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" Oct 31 04:09:32.373504 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.373471 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm" Oct 31 04:09:32.549054 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.548937 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5"] Oct 31 04:09:32.552374 ip-10-0-142-76 kubenswrapper[5490]: W1031 04:09:32.552341 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c20927_b1f8_4930_8a01_5a314fce9efc.slice/crio-92e8cf72dd3bcead6ee8e968502341ca11a019354b9755dab99beb48014eb80d WatchSource:0}: Error finding container 92e8cf72dd3bcead6ee8e968502341ca11a019354b9755dab99beb48014eb80d: Status 404 returned error can't find the container with id 92e8cf72dd3bcead6ee8e968502341ca11a019354b9755dab99beb48014eb80d Oct 31 04:09:32.553865 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.553844 5490 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:09:32.656646 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.656614 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm"] Oct 31 04:09:32.657995 ip-10-0-142-76 kubenswrapper[5490]: W1031 04:09:32.657966 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eeef949_c2c4_48e8_8099_ab62fd54f865.slice/crio-2c7f442845b26db515c11909633a83490ca7f37217b4e020a33f440f7369dce4 WatchSource:0}: Error finding container 2c7f442845b26db515c11909633a83490ca7f37217b4e020a33f440f7369dce4: Status 404 returned error can't find the container with id 2c7f442845b26db515c11909633a83490ca7f37217b4e020a33f440f7369dce4 Oct 31 04:09:32.839574 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:32.839469 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:33.520859 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:33.520817 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:33.535559 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:33.535519 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm" event={"ID":"9eeef949-c2c4-48e8-8099-ab62fd54f865","Type":"ContainerStarted","Data":"135498c7376f2deda0e33c4af8a93a638464b79af2a9c18183782045f354421e"} Oct 31 04:09:33.535781 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:33.535584 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm" event={"ID":"9eeef949-c2c4-48e8-8099-ab62fd54f865","Type":"ContainerStarted","Data":"2c7f442845b26db515c11909633a83490ca7f37217b4e020a33f440f7369dce4"} Oct 31 04:09:33.535781 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:33.535616 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm" Oct 31 04:09:33.537216 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:33.537183 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" event={"ID":"06c20927-b1f8-4930-8a01-5a314fce9efc","Type":"ContainerStarted","Data":"796b0d8ef9888e2e2ed61526982c5688e46e7f32bdcc622a03fa5d440f792a9a"} Oct 31 04:09:33.537336 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:33.537242 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" Oct 31 04:09:33.537336 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:33.537267 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" event={"ID":"06c20927-b1f8-4930-8a01-5a314fce9efc","Type":"ContainerStarted","Data":"92e8cf72dd3bcead6ee8e968502341ca11a019354b9755dab99beb48014eb80d"} Oct 31 04:09:33.552635 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:33.552582 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm" podStartSLOduration=1.552566409 podStartE2EDuration="1.552566409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:33.551397173 +0000 UTC m=+3208.086092129" watchObservedRunningTime="2026-10-31 04:09:33.552566409 +0000 UTC m=+3208.087261363" Oct 31 04:09:33.569315 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:33.569266 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" podStartSLOduration=2.027374533 podStartE2EDuration="2.569251558s" totalImagesPullingTime="541.877025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:33.567570369 +0000 UTC m=+3208.102265325" watchObservedRunningTime="2026-10-31 04:09:33.569251558 +0000 UTC m=+3208.103946513" Oct 31 04:09:34.211787 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:34.211731 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:35.227464 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:35.227420 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:35.904280 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:35.904237 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:36.609359 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:36.609325 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:37.225449 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:37.225401 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:37.957050 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:37.956943 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:38.625787 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:38.625744 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:39.361143 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:39.361104 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:39.555437 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:39.555406 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" Oct 31 04:09:40.247152 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:40.247017 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:41.105375 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:41.105337 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:41.802191 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:41.802140 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:42.553212 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:42.553175 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:43.258120 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:43.258081 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:43.931163 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:43.931122 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:44.514327 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:44.514288 5490 patch_prober.go:29] interesting pod/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 31 04:09:44.514525 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:44.514374 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" podUID="f28ec478-90c6-4623-8e9a-6d43b9311629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 31 04:09:44.546562 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:44.546532 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm" Oct 31 04:09:44.602573 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:44.602532 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:44.614814 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:44.614778 5490 generic.go:356] "Generic (PLEG): container finished" podID="f28ec478-90c6-4623-8e9a-6d43b9311629" containerID="68e7d903940736da4506d425d748a892c87b768e5e46e89977e115dacd2432bf" exitCode=0 Oct 31 04:09:44.614988 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:44.614841 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" event={"ID":"f28ec478-90c6-4623-8e9a-6d43b9311629","Type":"ContainerDied","Data":"68e7d903940736da4506d425d748a892c87b768e5e46e89977e115dacd2432bf"} Oct 31 04:09:44.615255 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:44.615241 5490 scope.go:122] "RemoveContainer" containerID="68e7d903940736da4506d425d748a892c87b768e5e46e89977e115dacd2432bf" Oct 31 04:09:44.616581 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:44.616548 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:09:44.616745 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:44.616640 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:09:44.616745 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:44.616723 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:09:45.494942 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:45.494902 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:45.622904 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:45.622857 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" event={"ID":"f28ec478-90c6-4623-8e9a-6d43b9311629","Type":"ContainerStarted","Data":"685abfbe8edf3c43ead6ef55e54867844086cd195fe914c1eac945b03376ea84"} Oct 31 04:09:45.734708 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:45.725979 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:46.107370 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:46.107328 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:46.255491 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:46.255451 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:46.256374 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:46.256338 5490 patch_prober.go:29] interesting pod/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 31 04:09:46.256569 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:46.256531 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" podUID="f28ec478-90c6-4623-8e9a-6d43b9311629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 31 04:09:46.279976 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:46.279928 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:46.434667 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:46.434629 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:47.088689 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:47.088637 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:47.160129 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:09:47.160084 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:09:47.256024 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:47.255986 5490 patch_prober.go:29] interesting pod/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 31 04:09:47.256214 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:47.256058 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" podUID="f28ec478-90c6-4623-8e9a-6d43b9311629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 31 04:09:47.286697 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:47.286637 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:47.922366 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:47.922323 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:48.255302 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:48.255204 5490 patch_prober.go:29] interesting pod/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 31 04:09:48.255302 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:48.255288 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" podUID="f28ec478-90c6-4623-8e9a-6d43b9311629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 31 04:09:48.647869 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:48.647778 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:49.255124 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:49.255082 5490 patch_prober.go:29] interesting pod/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 31 04:09:49.255311 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:49.255143 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" podUID="f28ec478-90c6-4623-8e9a-6d43b9311629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 31 04:09:49.515918 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:49.515809 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:50.237288 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:50.237249 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:50.255183 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:50.255145 5490 patch_prober.go:29] interesting pod/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 31 04:09:50.255400 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:50.255213 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" podUID="f28ec478-90c6-4623-8e9a-6d43b9311629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 31 04:09:51.154961 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:51.154921 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:51.259107 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:51.259076 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:51.259568 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:51.259556 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:51.260409 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:51.260384 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:09:51.803163 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:51.803126 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:52.508040 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:52.508000 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:53.108826 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:53.108791 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:53.713220 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:53.713178 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:54.311660 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:54.311618 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:55.003268 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:55.003182 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:55.722539 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:55.722495 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:56.307428 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:56.307388 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:56.985264 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:56.985218 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:57.594231 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:57.594184 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:58.186640 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:58.186597 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:58.787392 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:58.787349 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:09:59.591582 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:09:59.591543 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:00.201254 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:00.201218 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:00.784874 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:00.784836 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:01.422488 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:01.422457 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:02.148714 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:02.148633 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:02.795202 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:02.795159 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:03.499468 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:03.499431 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:04.409860 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:04.409807 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:05.094953 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:05.094906 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:05.905898 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:05.905848 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:06.571575 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:06.571533 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:07.289019 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:07.288908 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:07.892812 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:07.892753 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:08.511376 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:08.511336 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:09.148567 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:09.148356 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:09.795080 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:09.795043 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:10.488063 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:10.488017 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:11.098620 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:11.098583 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:11.711289 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:11.711229 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:12.313744 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:12.313703 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:12.903035 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:12.902985 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:13.617382 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:13.617337 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:14.224053 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:14.224012 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:14.891419 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:14.891361 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:15.492316 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:15.492273 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:16.090669 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:16.090623 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:16.697285 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:16.697245 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:17.128710 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:10:17.128611 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:10:17.305093 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:17.305006 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:18.000432 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:18.000383 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:18.700349 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:18.700309 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:19.038124 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:19.038029 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:19.454443 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:19.454404 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:20.096795 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:20.096736 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:20.789547 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:20.789511 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:21.408130 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:21.408083 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:22.104839 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:22.104802 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:22.819277 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:22.819235 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:23.517584 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:23.517542 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:25.224259 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:25.224222 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:25.363048 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:25.363006 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:25.621014 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:25.620913 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:26.307328 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:26.307288 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:27.103223 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:27.103190 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:27.813289 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:27.813250 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:28.706159 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:28.706123 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:28.962864 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:28.962772 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:29.505712 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:29.505636 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:30.298399 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:30.298362 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:31.002071 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:31.002027 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:31.617703 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:31.617651 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:32.306545 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:32.306505 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:32.978258 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:32.978196 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:33.629268 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:33.629227 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:33.748314 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:33.748272 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:34.300428 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:34.300376 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:34.889221 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:34.889186 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:35.483237 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:35.483204 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:36.195144 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:36.195107 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:36.790907 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:36.790855 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:37.393918 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:37.393882 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:37.996775 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:37.996536 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:38.639220 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:38.639184 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:38.871862 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:38.871825 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm"] Oct 31 04:10:38.996119 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:38.996087 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:39.130229 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:39.130195 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:39.298443 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:39.298307 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:39.710800 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:39.710761 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5"] Oct 31 04:10:39.711034 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:39.710984 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" podUID="06c20927-b1f8-4930-8a01-5a314fce9efc" containerName="maas-api" containerID="cri-o://796b0d8ef9888e2e2ed61526982c5688e46e7f32bdcc622a03fa5d440f792a9a" gracePeriod=30 Oct 31 04:10:39.817799 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:39.817761 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:40.060722 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.060684 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:40.171364 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.171338 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" Oct 31 04:10:40.205750 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.205707 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:40.247155 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.247071 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06c20927-b1f8-4930-8a01-5a314fce9efc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06c20927-b1f8-4930-8a01-5a314fce9efc-maas-api-tls\") pod \"06c20927-b1f8-4930-8a01-5a314fce9efc\" (UID: \"06c20927-b1f8-4930-8a01-5a314fce9efc\") " Oct 31 04:10:40.247155 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.247112 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06c20927-b1f8-4930-8a01-5a314fce9efc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c20927-b1f8-4930-8a01-5a314fce9efc-openshift-service-ca\") pod \"06c20927-b1f8-4930-8a01-5a314fce9efc\" (UID: \"06c20927-b1f8-4930-8a01-5a314fce9efc\") " Oct 31 04:10:40.247155 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.247147 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06c20927-b1f8-4930-8a01-5a314fce9efc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/06c20927-b1f8-4930-8a01-5a314fce9efc-metrics-certs\") pod \"06c20927-b1f8-4930-8a01-5a314fce9efc\" (UID: \"06c20927-b1f8-4930-8a01-5a314fce9efc\") " Oct 31 04:10:40.247472 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.247293 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c20927-b1f8-4930-8a01-5a314fce9efc-kube-api-access-x2l9s\" (UniqueName: \"kubernetes.io/projected/06c20927-b1f8-4930-8a01-5a314fce9efc-kube-api-access-x2l9s\") pod \"06c20927-b1f8-4930-8a01-5a314fce9efc\" (UID: \"06c20927-b1f8-4930-8a01-5a314fce9efc\") " Oct 31 04:10:40.247778 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.247614 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c20927-b1f8-4930-8a01-5a314fce9efc-openshift-service-ca" pod "06c20927-b1f8-4930-8a01-5a314fce9efc" (UID: "06c20927-b1f8-4930-8a01-5a314fce9efc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:10:40.249514 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.249489 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06c20927-b1f8-4930-8a01-5a314fce9efc-maas-api-tls" pod "06c20927-b1f8-4930-8a01-5a314fce9efc" (UID: "06c20927-b1f8-4930-8a01-5a314fce9efc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:10:40.249634 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.249607 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c20927-b1f8-4930-8a01-5a314fce9efc-kube-api-access-x2l9s" pod "06c20927-b1f8-4930-8a01-5a314fce9efc" (UID: "06c20927-b1f8-4930-8a01-5a314fce9efc"). InnerVolumeSpecName "kube-api-access-x2l9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:10:40.249712 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.249657 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06c20927-b1f8-4930-8a01-5a314fce9efc-metrics-certs" pod "06c20927-b1f8-4930-8a01-5a314fce9efc" (UID: "06c20927-b1f8-4930-8a01-5a314fce9efc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:10:40.348531 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.348493 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2l9s\" (UniqueName: \"kubernetes.io/projected/06c20927-b1f8-4930-8a01-5a314fce9efc-kube-api-access-x2l9s\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:40.348531 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.348525 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06c20927-b1f8-4930-8a01-5a314fce9efc-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:40.348531 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.348535 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06c20927-b1f8-4930-8a01-5a314fce9efc-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:40.348825 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.348545 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06c20927-b1f8-4930-8a01-5a314fce9efc-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:40.635976 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.635889 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:40.976926 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.976890 5490 generic.go:356] "Generic (PLEG): container finished" podID="06c20927-b1f8-4930-8a01-5a314fce9efc" containerID="796b0d8ef9888e2e2ed61526982c5688e46e7f32bdcc622a03fa5d440f792a9a" exitCode=0 Oct 31 04:10:40.976926 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.976934 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" Oct 31 04:10:40.977166 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.976974 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" event={"ID":"06c20927-b1f8-4930-8a01-5a314fce9efc","Type":"ContainerDied","Data":"796b0d8ef9888e2e2ed61526982c5688e46e7f32bdcc622a03fa5d440f792a9a"} Oct 31 04:10:40.977166 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.977034 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5" event={"ID":"06c20927-b1f8-4930-8a01-5a314fce9efc","Type":"ContainerDied","Data":"92e8cf72dd3bcead6ee8e968502341ca11a019354b9755dab99beb48014eb80d"} Oct 31 04:10:40.977166 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.977058 5490 scope.go:122] "RemoveContainer" containerID="796b0d8ef9888e2e2ed61526982c5688e46e7f32bdcc622a03fa5d440f792a9a" Oct 31 04:10:40.987670 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.987647 5490 scope.go:122] "RemoveContainer" containerID="796b0d8ef9888e2e2ed61526982c5688e46e7f32bdcc622a03fa5d440f792a9a" Oct 31 04:10:40.987992 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:10:40.987968 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"796b0d8ef9888e2e2ed61526982c5688e46e7f32bdcc622a03fa5d440f792a9a\": container with ID starting with 796b0d8ef9888e2e2ed61526982c5688e46e7f32bdcc622a03fa5d440f792a9a not found: ID does not exist" containerID="796b0d8ef9888e2e2ed61526982c5688e46e7f32bdcc622a03fa5d440f792a9a" Oct 31 04:10:40.988074 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:40.988007 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"796b0d8ef9888e2e2ed61526982c5688e46e7f32bdcc622a03fa5d440f792a9a"} err="failed to get container status \"796b0d8ef9888e2e2ed61526982c5688e46e7f32bdcc622a03fa5d440f792a9a\": rpc error: code = NotFound desc = could not find container \"796b0d8ef9888e2e2ed61526982c5688e46e7f32bdcc622a03fa5d440f792a9a\": container with ID starting with 796b0d8ef9888e2e2ed61526982c5688e46e7f32bdcc622a03fa5d440f792a9a not found: ID does not exist" Oct 31 04:10:41.000628 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:41.000600 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5"] Oct 31 04:10:41.005958 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:41.005930 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8a336e-79755877bb-97qn5"] Oct 31 04:10:41.575277 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:41.575244 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06c20927-b1f8-4930-8a01-5a314fce9efc" path="/var/lib/kubelet/pods/06c20927-b1f8-4930-8a01-5a314fce9efc/volumes" Oct 31 04:10:43.872639 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:43.872594 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm" podUID="9eeef949-c2c4-48e8-8099-ab62fd54f865" containerName="payload-processing" containerID="cri-o://135498c7376f2deda0e33c4af8a93a638464b79af2a9c18183782045f354421e" gracePeriod=25 Oct 31 04:10:44.075158 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:44.075125 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:44.544159 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:44.544114 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm" podUID="9eeef949-c2c4-48e8-8099-ab62fd54f865" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:10:46.831989 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:46.831931 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:46.968396 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:46.968354 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:47.139772 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:10:47.139648 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:10:47.213145 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:47.213105 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp"] Oct 31 04:10:47.213350 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:47.213311 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" podUID="f28ec478-90c6-4623-8e9a-6d43b9311629" containerName="istio-proxy" containerID="cri-o://685abfbe8edf3c43ead6ef55e54867844086cd195fe914c1eac945b03376ea84" gracePeriod=30 Oct 31 04:10:47.307025 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:47.306980 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:51.262316 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:51.262275 5490 patch_prober.go:29] interesting pod/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 04:10:51.262810 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:51.262342 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" podUID="f28ec478-90c6-4623-8e9a-6d43b9311629" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 04:10:51.641101 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:51.640990 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:10:52.466149 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.466120 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:10:52.477028 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.476963 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-workload-certs\") pod \"f28ec478-90c6-4623-8e9a-6d43b9311629\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " Oct 31 04:10:52.477028 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.477008 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-data\") pod \"f28ec478-90c6-4623-8e9a-6d43b9311629\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " Oct 31 04:10:52.477235 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.477044 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28ec478-90c6-4623-8e9a-6d43b9311629-kube-api-access-xh9tv\" (UniqueName: \"kubernetes.io/projected/f28ec478-90c6-4623-8e9a-6d43b9311629-kube-api-access-xh9tv\") pod \"f28ec478-90c6-4623-8e9a-6d43b9311629\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " Oct 31 04:10:52.477235 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.477075 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-token\" (UniqueName: \"kubernetes.io/projected/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-token\") pod \"f28ec478-90c6-4623-8e9a-6d43b9311629\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " Oct 31 04:10:52.477235 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.477120 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f28ec478-90c6-4623-8e9a-6d43b9311629-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28ec478-90c6-4623-8e9a-6d43b9311629-istiod-ca-cert\") pod \"f28ec478-90c6-4623-8e9a-6d43b9311629\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " Oct 31 04:10:52.477235 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.477159 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-workload-socket\") pod \"f28ec478-90c6-4623-8e9a-6d43b9311629\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " Oct 31 04:10:52.477235 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.477210 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-podinfo\") pod \"f28ec478-90c6-4623-8e9a-6d43b9311629\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " Oct 31 04:10:52.477235 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.477219 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-workload-certs" pod "f28ec478-90c6-4623-8e9a-6d43b9311629" (UID: "f28ec478-90c6-4623-8e9a-6d43b9311629"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:10:52.477564 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.477262 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-credential-socket\") pod \"f28ec478-90c6-4623-8e9a-6d43b9311629\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " Oct 31 04:10:52.477564 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.477318 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-envoy\") pod \"f28ec478-90c6-4623-8e9a-6d43b9311629\" (UID: \"f28ec478-90c6-4623-8e9a-6d43b9311629\") " Oct 31 04:10:52.477564 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.477335 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-data" pod "f28ec478-90c6-4623-8e9a-6d43b9311629" (UID: "f28ec478-90c6-4623-8e9a-6d43b9311629"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:10:52.477737 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.477553 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-workload-socket" pod "f28ec478-90c6-4623-8e9a-6d43b9311629" (UID: "f28ec478-90c6-4623-8e9a-6d43b9311629"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:10:52.477737 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.477588 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-credential-socket" pod "f28ec478-90c6-4623-8e9a-6d43b9311629" (UID: "f28ec478-90c6-4623-8e9a-6d43b9311629"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:10:52.477737 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.477621 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f28ec478-90c6-4623-8e9a-6d43b9311629-istiod-ca-cert" pod "f28ec478-90c6-4623-8e9a-6d43b9311629" (UID: "f28ec478-90c6-4623-8e9a-6d43b9311629"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:10:52.477881 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.477755 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28ec478-90c6-4623-8e9a-6d43b9311629-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:52.477881 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.477776 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:52.477881 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.477792 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:52.477881 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.477805 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:52.477881 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.477825 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:52.479472 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.479430 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-podinfo" pod "f28ec478-90c6-4623-8e9a-6d43b9311629" (UID: "f28ec478-90c6-4623-8e9a-6d43b9311629"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:10:52.479586 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.479494 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-envoy" pod "f28ec478-90c6-4623-8e9a-6d43b9311629" (UID: "f28ec478-90c6-4623-8e9a-6d43b9311629"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:10:52.479766 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.479747 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-token" pod "f28ec478-90c6-4623-8e9a-6d43b9311629" (UID: "f28ec478-90c6-4623-8e9a-6d43b9311629"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:10:52.479829 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.479760 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28ec478-90c6-4623-8e9a-6d43b9311629-kube-api-access-xh9tv" pod "f28ec478-90c6-4623-8e9a-6d43b9311629" (UID: "f28ec478-90c6-4623-8e9a-6d43b9311629"). InnerVolumeSpecName "kube-api-access-xh9tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:10:52.578974 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.578936 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh9tv\" (UniqueName: \"kubernetes.io/projected/f28ec478-90c6-4623-8e9a-6d43b9311629-kube-api-access-xh9tv\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:52.578974 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.578972 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:52.579184 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.578986 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:52.579184 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:52.578995 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28ec478-90c6-4623-8e9a-6d43b9311629-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:10:53.049047 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:53.049014 5490 generic.go:356] "Generic (PLEG): container finished" podID="f28ec478-90c6-4623-8e9a-6d43b9311629" containerID="685abfbe8edf3c43ead6ef55e54867844086cd195fe914c1eac945b03376ea84" exitCode=0 Oct 31 04:10:53.049243 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:53.049075 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" Oct 31 04:10:53.049243 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:53.049105 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" event={"ID":"f28ec478-90c6-4623-8e9a-6d43b9311629","Type":"ContainerDied","Data":"685abfbe8edf3c43ead6ef55e54867844086cd195fe914c1eac945b03376ea84"} Oct 31 04:10:53.049243 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:53.049163 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp" event={"ID":"f28ec478-90c6-4623-8e9a-6d43b9311629","Type":"ContainerDied","Data":"761dcf276cf6ff1a155d165375f3e2babc0e8229f8833cd99cc9058a1ac7713a"} Oct 31 04:10:53.049243 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:53.049190 5490 scope.go:122] "RemoveContainer" containerID="685abfbe8edf3c43ead6ef55e54867844086cd195fe914c1eac945b03376ea84" Oct 31 04:10:53.059436 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:53.059416 5490 scope.go:122] "RemoveContainer" containerID="68e7d903940736da4506d425d748a892c87b768e5e46e89977e115dacd2432bf" Oct 31 04:10:53.068503 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:53.068479 5490 scope.go:122] "RemoveContainer" containerID="685abfbe8edf3c43ead6ef55e54867844086cd195fe914c1eac945b03376ea84" Oct 31 04:10:53.068808 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:10:53.068789 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685abfbe8edf3c43ead6ef55e54867844086cd195fe914c1eac945b03376ea84\": container with ID starting with 685abfbe8edf3c43ead6ef55e54867844086cd195fe914c1eac945b03376ea84 not found: ID does not exist" containerID="685abfbe8edf3c43ead6ef55e54867844086cd195fe914c1eac945b03376ea84" Oct 31 04:10:53.068865 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:53.068819 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685abfbe8edf3c43ead6ef55e54867844086cd195fe914c1eac945b03376ea84"} err="failed to get container status \"685abfbe8edf3c43ead6ef55e54867844086cd195fe914c1eac945b03376ea84\": rpc error: code = NotFound desc = could not find container \"685abfbe8edf3c43ead6ef55e54867844086cd195fe914c1eac945b03376ea84\": container with ID starting with 685abfbe8edf3c43ead6ef55e54867844086cd195fe914c1eac945b03376ea84 not found: ID does not exist" Oct 31 04:10:53.068865 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:53.068838 5490 scope.go:122] "RemoveContainer" containerID="68e7d903940736da4506d425d748a892c87b768e5e46e89977e115dacd2432bf" Oct 31 04:10:53.069102 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:10:53.069083 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68e7d903940736da4506d425d748a892c87b768e5e46e89977e115dacd2432bf\": container with ID starting with 68e7d903940736da4506d425d748a892c87b768e5e46e89977e115dacd2432bf not found: ID does not exist" containerID="68e7d903940736da4506d425d748a892c87b768e5e46e89977e115dacd2432bf" Oct 31 04:10:53.069159 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:53.069112 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e7d903940736da4506d425d748a892c87b768e5e46e89977e115dacd2432bf"} err="failed to get container status \"68e7d903940736da4506d425d748a892c87b768e5e46e89977e115dacd2432bf\": rpc error: code = NotFound desc = could not find container \"68e7d903940736da4506d425d748a892c87b768e5e46e89977e115dacd2432bf\": container with ID starting with 68e7d903940736da4506d425d748a892c87b768e5e46e89977e115dacd2432bf not found: ID does not exist" Oct 31 04:10:53.074402 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:53.074375 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp"] Oct 31 04:10:53.080122 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:53.080099 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8a336e-openshift-default-6b5686b9d5q7jmp"] Oct 31 04:10:53.575949 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:53.575917 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f28ec478-90c6-4623-8e9a-6d43b9311629" path="/var/lib/kubelet/pods/f28ec478-90c6-4623-8e9a-6d43b9311629/volumes" Oct 31 04:10:54.543839 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:54.543794 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm" podUID="9eeef949-c2c4-48e8-8099-ab62fd54f865" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 04:10:55.896863 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:55.896819 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-rssq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:10:55.897530 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:55.897495 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-rssq8" podUID="8eae7b8f-98ca-4de7-92e0-ff47b28d6927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:10:55.947873 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:55.947834 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-svnrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:10:55.948228 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:10:55.948176 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" podUID="b3c2974d-83ad-4321-93eb-60ec9738d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:11:04.119424 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:04.119397 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm" Oct 31 04:11:04.146773 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:04.146696 5490 generic.go:356] "Generic (PLEG): container finished" podID="9eeef949-c2c4-48e8-8099-ab62fd54f865" containerID="135498c7376f2deda0e33c4af8a93a638464b79af2a9c18183782045f354421e" exitCode=0 Oct 31 04:11:04.146773 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:04.146734 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm" Oct 31 04:11:04.146957 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:04.146774 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm" event={"ID":"9eeef949-c2c4-48e8-8099-ab62fd54f865","Type":"ContainerDied","Data":"135498c7376f2deda0e33c4af8a93a638464b79af2a9c18183782045f354421e"} Oct 31 04:11:04.146957 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:04.146830 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm" event={"ID":"9eeef949-c2c4-48e8-8099-ab62fd54f865","Type":"ContainerDied","Data":"2c7f442845b26db515c11909633a83490ca7f37217b4e020a33f440f7369dce4"} Oct 31 04:11:04.146957 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:04.146851 5490 scope.go:122] "RemoveContainer" containerID="135498c7376f2deda0e33c4af8a93a638464b79af2a9c18183782045f354421e" Oct 31 04:11:04.158177 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:04.158144 5490 scope.go:122] "RemoveContainer" containerID="135498c7376f2deda0e33c4af8a93a638464b79af2a9c18183782045f354421e" Oct 31 04:11:04.158512 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:11:04.158484 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"135498c7376f2deda0e33c4af8a93a638464b79af2a9c18183782045f354421e\": container with ID starting with 135498c7376f2deda0e33c4af8a93a638464b79af2a9c18183782045f354421e not found: ID does not exist" containerID="135498c7376f2deda0e33c4af8a93a638464b79af2a9c18183782045f354421e" Oct 31 04:11:04.158630 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:04.158520 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"135498c7376f2deda0e33c4af8a93a638464b79af2a9c18183782045f354421e"} err="failed to get container status \"135498c7376f2deda0e33c4af8a93a638464b79af2a9c18183782045f354421e\": rpc error: code = NotFound desc = could not find container \"135498c7376f2deda0e33c4af8a93a638464b79af2a9c18183782045f354421e\": container with ID starting with 135498c7376f2deda0e33c4af8a93a638464b79af2a9c18183782045f354421e not found: ID does not exist" Oct 31 04:11:04.204521 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:04.204483 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eeef949-c2c4-48e8-8099-ab62fd54f865-kube-api-access-5dbsf\" (UniqueName: \"kubernetes.io/projected/9eeef949-c2c4-48e8-8099-ab62fd54f865-kube-api-access-5dbsf\") pod \"9eeef949-c2c4-48e8-8099-ab62fd54f865\" (UID: \"9eeef949-c2c4-48e8-8099-ab62fd54f865\") " Oct 31 04:11:04.204716 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:04.204581 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eeef949-c2c4-48e8-8099-ab62fd54f865-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eeef949-c2c4-48e8-8099-ab62fd54f865-plugins-config-volume\") pod \"9eeef949-c2c4-48e8-8099-ab62fd54f865\" (UID: \"9eeef949-c2c4-48e8-8099-ab62fd54f865\") " Oct 31 04:11:04.204974 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:04.204950 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eeef949-c2c4-48e8-8099-ab62fd54f865-plugins-config-volume" pod "9eeef949-c2c4-48e8-8099-ab62fd54f865" (UID: "9eeef949-c2c4-48e8-8099-ab62fd54f865"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:11:04.206631 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:04.206605 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eeef949-c2c4-48e8-8099-ab62fd54f865-kube-api-access-5dbsf" pod "9eeef949-c2c4-48e8-8099-ab62fd54f865" (UID: "9eeef949-c2c4-48e8-8099-ab62fd54f865"). InnerVolumeSpecName "kube-api-access-5dbsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:04.306082 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:04.306043 5490 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eeef949-c2c4-48e8-8099-ab62fd54f865-plugins-config-volume\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:04.306082 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:04.306075 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dbsf\" (UniqueName: \"kubernetes.io/projected/9eeef949-c2c4-48e8-8099-ab62fd54f865-kube-api-access-5dbsf\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:04.469926 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:04.469886 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm"] Oct 31 04:11:04.472436 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:04.472410 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8a336e-58b4678cfgxmfm"] Oct 31 04:11:05.574215 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:05.574183 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eeef949-c2c4-48e8-8099-ab62fd54f865" path="/var/lib/kubelet/pods/9eeef949-c2c4-48e8-8099-ab62fd54f865/volumes" Oct 31 04:11:07.521474 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.521439 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp"] Oct 31 04:11:07.521952 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.521937 5490 cpu_manager.go:414] "containerMap: removing container" podUID="06c20927-b1f8-4930-8a01-5a314fce9efc" containerName="maas-api" Oct 31 04:11:07.521999 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.521953 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c20927-b1f8-4930-8a01-5a314fce9efc" containerName="maas-api" Oct 31 04:11:07.521999 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.521971 5490 cpu_manager.go:414] "containerMap: removing container" podUID="f28ec478-90c6-4623-8e9a-6d43b9311629" containerName="istio-proxy" Oct 31 04:11:07.521999 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.521977 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28ec478-90c6-4623-8e9a-6d43b9311629" containerName="istio-proxy" Oct 31 04:11:07.521999 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.521991 5490 cpu_manager.go:414] "containerMap: removing container" podUID="f28ec478-90c6-4623-8e9a-6d43b9311629" containerName="istio-proxy" Oct 31 04:11:07.521999 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.521999 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28ec478-90c6-4623-8e9a-6d43b9311629" containerName="istio-proxy" Oct 31 04:11:07.522158 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.522012 5490 cpu_manager.go:414] "containerMap: removing container" podUID="9eeef949-c2c4-48e8-8099-ab62fd54f865" containerName="payload-processing" Oct 31 04:11:07.522158 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.522017 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eeef949-c2c4-48e8-8099-ab62fd54f865" containerName="payload-processing" Oct 31 04:11:07.522158 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.522089 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eeef949-c2c4-48e8-8099-ab62fd54f865" containerName="payload-processing" Oct 31 04:11:07.522158 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.522105 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c20927-b1f8-4930-8a01-5a314fce9efc" containerName="maas-api" Oct 31 04:11:07.522158 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.522113 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28ec478-90c6-4623-8e9a-6d43b9311629" containerName="istio-proxy" Oct 31 04:11:07.522158 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.522120 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28ec478-90c6-4623-8e9a-6d43b9311629" containerName="istio-proxy" Oct 31 04:11:07.525282 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.525263 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:07.533966 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.533548 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-85f4c8a5-kserve-self-signed-certs\"" Oct 31 04:11:07.535575 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.535542 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp"] Oct 31 04:11:07.625351 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.625315 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:07.640367 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.640326 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c22c27a-c74b-402b-a232-dc6f5063223e-tls-certs\") pod \"e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:07.640519 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.640392 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-dshm\") pod \"e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:07.640566 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.640528 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff7zr\" (UniqueName: \"kubernetes.io/projected/3c22c27a-c74b-402b-a232-dc6f5063223e-kube-api-access-ff7zr\") pod \"e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:07.640615 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.640563 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-tmp-dir\") pod \"e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:07.640615 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.640597 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-model-cache\") pod \"e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:07.640728 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.640668 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-home\") pod \"e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:07.742055 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.742013 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-home\") pod \"e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:07.742229 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.742077 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c22c27a-c74b-402b-a232-dc6f5063223e-tls-certs\") pod \"e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:07.742229 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.742125 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-dshm\") pod \"e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:07.742325 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.742257 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff7zr\" (UniqueName: \"kubernetes.io/projected/3c22c27a-c74b-402b-a232-dc6f5063223e-kube-api-access-ff7zr\") pod \"e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:07.742325 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.742308 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-tmp-dir\") pod \"e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:07.742428 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.742344 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-model-cache\") pod \"e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:07.742486 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.742435 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-home\") pod \"e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:07.742716 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.742660 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-tmp-dir\") pod \"e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:07.742859 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.742749 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-model-cache\") pod \"e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:07.744503 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.744479 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-dshm\") pod \"e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:07.744799 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.744781 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c22c27a-c74b-402b-a232-dc6f5063223e-tls-certs\") pod \"e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:07.749557 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.749539 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff7zr\" (UniqueName: \"kubernetes.io/projected/3c22c27a-c74b-402b-a232-dc6f5063223e-kube-api-access-ff7zr\") pod \"e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:07.840007 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:07.839923 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:08.117100 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:08.117075 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp"] Oct 31 04:11:08.119559 ip-10-0-142-76 kubenswrapper[5490]: W1031 04:11:08.119521 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c22c27a_c74b_402b_a232_dc6f5063223e.slice/crio-4172798e0a8668d9bebe2cb82373db588a7b42f60b950d59d507159381430347 WatchSource:0}: Error finding container 4172798e0a8668d9bebe2cb82373db588a7b42f60b950d59d507159381430347: Status 404 returned error can't find the container with id 4172798e0a8668d9bebe2cb82373db588a7b42f60b950d59d507159381430347 Oct 31 04:11:08.171475 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:08.171444 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" event={"ID":"3c22c27a-c74b-402b-a232-dc6f5063223e","Type":"ContainerStarted","Data":"4172798e0a8668d9bebe2cb82373db588a7b42f60b950d59d507159381430347"} Oct 31 04:11:09.179545 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:09.179500 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" event={"ID":"3c22c27a-c74b-402b-a232-dc6f5063223e","Type":"ContainerStarted","Data":"da3fc5fb354f5c2ff8b6a56b632451752526a0c377cc14987ba4e6bf6d6f4849"} Oct 31 04:11:09.200238 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:09.200180 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" podStartSLOduration=2.200165761 podStartE2EDuration="2.200165761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:09.198038957 +0000 UTC m=+3303.732733914" watchObservedRunningTime="2026-10-31 04:11:09.200165761 +0000 UTC m=+3303.734860727" Oct 31 04:11:09.840367 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:09.840319 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:16.201045 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:16.201003 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:17.127131 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:11:17.127092 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:11:17.177983 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:17.177934 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:17.841063 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:17.841030 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:17.854608 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:17.854579 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:18.252029 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:18.251997 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:19.074105 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:19.074071 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:19.199738 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:19.199701 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:22.633613 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:22.633579 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:35.610387 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:35.610343 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:36.318933 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:36.318893 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp"] Oct 31 04:11:36.397896 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:36.397860 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:11:47.126967 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:11:47.126869 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:11:51.341053 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.340988 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" podUID="3c22c27a-c74b-402b-a232-dc6f5063223e" containerName="main" containerID="cri-o://da3fc5fb354f5c2ff8b6a56b632451752526a0c377cc14987ba4e6bf6d6f4849" gracePeriod=45 Oct 31 04:11:51.492924 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.492871 5490 generic.go:356] "Generic (PLEG): container finished" podID="3c22c27a-c74b-402b-a232-dc6f5063223e" containerID="da3fc5fb354f5c2ff8b6a56b632451752526a0c377cc14987ba4e6bf6d6f4849" exitCode=0 Oct 31 04:11:51.493117 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.492945 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" event={"ID":"3c22c27a-c74b-402b-a232-dc6f5063223e","Type":"ContainerDied","Data":"da3fc5fb354f5c2ff8b6a56b632451752526a0c377cc14987ba4e6bf6d6f4849"} Oct 31 04:11:51.654936 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.654907 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:51.716078 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.716036 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c22c27a-c74b-402b-a232-dc6f5063223e-kube-api-access-ff7zr\" (UniqueName: \"kubernetes.io/projected/3c22c27a-c74b-402b-a232-dc6f5063223e-kube-api-access-ff7zr\") pod \"3c22c27a-c74b-402b-a232-dc6f5063223e\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " Oct 31 04:11:51.716296 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.716095 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-dshm\") pod \"3c22c27a-c74b-402b-a232-dc6f5063223e\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " Oct 31 04:11:51.716296 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.716162 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-tmp-dir\") pod \"3c22c27a-c74b-402b-a232-dc6f5063223e\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " Oct 31 04:11:51.716296 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.716208 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-home\" (UniqueName: \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-home\") pod \"3c22c27a-c74b-402b-a232-dc6f5063223e\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " Oct 31 04:11:51.716296 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.716235 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c22c27a-c74b-402b-a232-dc6f5063223e-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c22c27a-c74b-402b-a232-dc6f5063223e-tls-certs\") pod \"3c22c27a-c74b-402b-a232-dc6f5063223e\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " Oct 31 04:11:51.716296 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.716282 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-model-cache\") pod \"3c22c27a-c74b-402b-a232-dc6f5063223e\" (UID: \"3c22c27a-c74b-402b-a232-dc6f5063223e\") " Oct 31 04:11:51.716744 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.716713 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-tmp-dir" pod "3c22c27a-c74b-402b-a232-dc6f5063223e" (UID: "3c22c27a-c74b-402b-a232-dc6f5063223e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:51.716854 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.716825 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-model-cache" pod "3c22c27a-c74b-402b-a232-dc6f5063223e" (UID: "3c22c27a-c74b-402b-a232-dc6f5063223e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:51.717074 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.717047 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-home" pod "3c22c27a-c74b-402b-a232-dc6f5063223e" (UID: "3c22c27a-c74b-402b-a232-dc6f5063223e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:51.719109 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.719068 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c22c27a-c74b-402b-a232-dc6f5063223e-kube-api-access-ff7zr" pod "3c22c27a-c74b-402b-a232-dc6f5063223e" (UID: "3c22c27a-c74b-402b-a232-dc6f5063223e"). InnerVolumeSpecName "kube-api-access-ff7zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:11:51.719656 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.719622 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c22c27a-c74b-402b-a232-dc6f5063223e-tls-certs" pod "3c22c27a-c74b-402b-a232-dc6f5063223e" (UID: "3c22c27a-c74b-402b-a232-dc6f5063223e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:11:51.720075 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.720039 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-dshm" pod "3c22c27a-c74b-402b-a232-dc6f5063223e" (UID: "3c22c27a-c74b-402b-a232-dc6f5063223e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:11:51.817958 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.817893 5490 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-tmp-dir\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:51.817958 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.817951 5490 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-home\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:51.817958 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.817968 5490 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c22c27a-c74b-402b-a232-dc6f5063223e-tls-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:51.818320 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.817982 5490 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-model-cache\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:51.818320 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.817999 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff7zr\" (UniqueName: \"kubernetes.io/projected/3c22c27a-c74b-402b-a232-dc6f5063223e-kube-api-access-ff7zr\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:51.818320 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:51.818013 5490 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c22c27a-c74b-402b-a232-dc6f5063223e-dshm\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:11:52.506698 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:52.506569 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" Oct 31 04:11:52.506698 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:52.506588 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp" event={"ID":"3c22c27a-c74b-402b-a232-dc6f5063223e","Type":"ContainerDied","Data":"4172798e0a8668d9bebe2cb82373db588a7b42f60b950d59d507159381430347"} Oct 31 04:11:52.506698 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:52.506643 5490 scope.go:122] "RemoveContainer" containerID="da3fc5fb354f5c2ff8b6a56b632451752526a0c377cc14987ba4e6bf6d6f4849" Oct 31 04:11:52.529867 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:52.529833 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp"] Oct 31 04:11:52.550701 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:52.548589 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-85f4c8a5-kserve-65d498bbf7-xn8lp"] Oct 31 04:11:53.593724 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:11:53.590564 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c22c27a-c74b-402b-a232-dc6f5063223e" path="/var/lib/kubelet/pods/3c22c27a-c74b-402b-a232-dc6f5063223e/volumes" Oct 31 04:12:10.297653 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:10.297615 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bf8887774-hr8xq"] Oct 31 04:12:10.298324 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:10.298304 5490 cpu_manager.go:414] "containerMap: removing container" podUID="3c22c27a-c74b-402b-a232-dc6f5063223e" containerName="main" Oct 31 04:12:10.298417 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:10.298326 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c22c27a-c74b-402b-a232-dc6f5063223e" containerName="main" Oct 31 04:12:10.298478 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:10.298467 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c22c27a-c74b-402b-a232-dc6f5063223e" containerName="main" Oct 31 04:12:10.302061 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:10.302037 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bf8887774-hr8xq" Oct 31 04:12:10.313020 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:10.312990 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bf8887774-hr8xq"] Oct 31 04:12:10.434201 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:10.434153 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpbg8\" (UniqueName: \"kubernetes.io/projected/7cb9f5f1-55a1-4702-a299-67cae7e36032-kube-api-access-fpbg8\") pod \"maas-api-bf8887774-hr8xq\" (UID: \"7cb9f5f1-55a1-4702-a299-67cae7e36032\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-hr8xq" Oct 31 04:12:10.434201 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:10.434208 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb9f5f1-55a1-4702-a299-67cae7e36032-openshift-service-ca\") pod \"maas-api-bf8887774-hr8xq\" (UID: \"7cb9f5f1-55a1-4702-a299-67cae7e36032\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-hr8xq" Oct 31 04:12:10.434498 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:10.434313 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cb9f5f1-55a1-4702-a299-67cae7e36032-metrics-certs\") pod \"maas-api-bf8887774-hr8xq\" (UID: \"7cb9f5f1-55a1-4702-a299-67cae7e36032\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-hr8xq" Oct 31 04:12:10.434498 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:10.434354 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cb9f5f1-55a1-4702-a299-67cae7e36032-maas-api-tls\") pod \"maas-api-bf8887774-hr8xq\" (UID: \"7cb9f5f1-55a1-4702-a299-67cae7e36032\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-hr8xq" Oct 31 04:12:10.535854 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:10.535812 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpbg8\" (UniqueName: \"kubernetes.io/projected/7cb9f5f1-55a1-4702-a299-67cae7e36032-kube-api-access-fpbg8\") pod \"maas-api-bf8887774-hr8xq\" (UID: \"7cb9f5f1-55a1-4702-a299-67cae7e36032\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-hr8xq" Oct 31 04:12:10.535854 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:10.535858 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb9f5f1-55a1-4702-a299-67cae7e36032-openshift-service-ca\") pod \"maas-api-bf8887774-hr8xq\" (UID: \"7cb9f5f1-55a1-4702-a299-67cae7e36032\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-hr8xq" Oct 31 04:12:10.536083 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:10.535890 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cb9f5f1-55a1-4702-a299-67cae7e36032-metrics-certs\") pod \"maas-api-bf8887774-hr8xq\" (UID: \"7cb9f5f1-55a1-4702-a299-67cae7e36032\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-hr8xq" Oct 31 04:12:10.536083 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:10.535931 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cb9f5f1-55a1-4702-a299-67cae7e36032-maas-api-tls\") pod \"maas-api-bf8887774-hr8xq\" (UID: \"7cb9f5f1-55a1-4702-a299-67cae7e36032\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-hr8xq" Oct 31 04:12:10.536549 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:10.536521 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb9f5f1-55a1-4702-a299-67cae7e36032-openshift-service-ca\") pod \"maas-api-bf8887774-hr8xq\" (UID: \"7cb9f5f1-55a1-4702-a299-67cae7e36032\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-hr8xq" Oct 31 04:12:10.538456 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:10.538429 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cb9f5f1-55a1-4702-a299-67cae7e36032-maas-api-tls\") pod \"maas-api-bf8887774-hr8xq\" (UID: \"7cb9f5f1-55a1-4702-a299-67cae7e36032\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-hr8xq" Oct 31 04:12:10.538572 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:10.538478 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cb9f5f1-55a1-4702-a299-67cae7e36032-metrics-certs\") pod \"maas-api-bf8887774-hr8xq\" (UID: \"7cb9f5f1-55a1-4702-a299-67cae7e36032\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-hr8xq" Oct 31 04:12:10.542963 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:10.542939 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpbg8\" (UniqueName: \"kubernetes.io/projected/7cb9f5f1-55a1-4702-a299-67cae7e36032-kube-api-access-fpbg8\") pod \"maas-api-bf8887774-hr8xq\" (UID: \"7cb9f5f1-55a1-4702-a299-67cae7e36032\") " pod="odh-ai-gateway-infra/maas-api-bf8887774-hr8xq" Oct 31 04:12:10.621311 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:10.621222 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bf8887774-hr8xq" Oct 31 04:12:10.897666 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:10.897641 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bf8887774-hr8xq"] Oct 31 04:12:10.899438 ip-10-0-142-76 kubenswrapper[5490]: W1031 04:12:10.899409 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb9f5f1_55a1_4702_a299_67cae7e36032.slice/crio-3b6fd949498f3c609015e304d1098fc0e267a2142ef3c0a56f1b79530933cee7 WatchSource:0}: Error finding container 3b6fd949498f3c609015e304d1098fc0e267a2142ef3c0a56f1b79530933cee7: Status 404 returned error can't find the container with id 3b6fd949498f3c609015e304d1098fc0e267a2142ef3c0a56f1b79530933cee7 Oct 31 04:12:11.616614 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:11.616577 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bf8887774-hr8xq" event={"ID":"7cb9f5f1-55a1-4702-a299-67cae7e36032","Type":"ContainerStarted","Data":"139c222f6ddad9a96822c99b58d7cefabca60940c8d934d3907630d8481c9244"} Oct 31 04:12:11.617131 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:11.616635 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bf8887774-hr8xq" event={"ID":"7cb9f5f1-55a1-4702-a299-67cae7e36032","Type":"ContainerStarted","Data":"3b6fd949498f3c609015e304d1098fc0e267a2142ef3c0a56f1b79530933cee7"} Oct 31 04:12:11.617131 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:11.616727 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bf8887774-hr8xq" Oct 31 04:12:11.635535 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:11.635478 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bf8887774-hr8xq" podStartSLOduration=1.069269377 podStartE2EDuration="1.635464286s" totalImagesPullingTime="566.194909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:11.634180091 +0000 UTC m=+3366.168875049" watchObservedRunningTime="2026-10-31 04:12:11.635464286 +0000 UTC m=+3366.170159256" Oct 31 04:12:17.119933 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.119886 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:17.127379 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:12:17.127351 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:12:17.461068 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.461029 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv"] Oct 31 04:12:17.465307 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.465278 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.469629 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.469603 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4bd78aa5-openshift-default-dockercfg-x6p8q\"" Oct 31 04:12:17.478341 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.478305 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv"] Oct 31 04:12:17.594100 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.594067 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:17.613565 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.613534 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-workload-socket\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.613565 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.613570 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-credential-socket\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.613876 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.613593 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-podinfo\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.613876 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.613751 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-token\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.613966 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.613922 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-envoy\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.614025 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.613969 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-workload-certs\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.614127 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.614110 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-data\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.614199 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.614147 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d77f97d-a54d-4e24-a128-e15fa735e656-istiod-ca-cert\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.614307 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.614288 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj5dz\" (UniqueName: \"kubernetes.io/projected/3d77f97d-a54d-4e24-a128-e15fa735e656-kube-api-access-qj5dz\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.628700 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.628667 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bf8887774-hr8xq" Oct 31 04:12:17.715403 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.715298 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj5dz\" (UniqueName: \"kubernetes.io/projected/3d77f97d-a54d-4e24-a128-e15fa735e656-kube-api-access-qj5dz\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.715403 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.715376 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-workload-socket\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.715670 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.715421 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-credential-socket\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.715670 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.715450 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-podinfo\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.715670 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.715479 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-token\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.715670 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.715531 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-envoy\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.715670 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.715564 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-workload-certs\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.715994 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.715800 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-data\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.715994 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.715839 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d77f97d-a54d-4e24-a128-e15fa735e656-istiod-ca-cert\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.716234 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.716210 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-credential-socket\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.716431 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.716406 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-workload-certs\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.716642 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.715837 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-workload-socket\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.717242 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.717217 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-data\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.717658 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.717631 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d77f97d-a54d-4e24-a128-e15fa735e656-istiod-ca-cert\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.718778 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.718749 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-podinfo\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.719154 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.719132 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-envoy\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.730843 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.730805 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-token\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.731099 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.731073 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj5dz\" (UniqueName: \"kubernetes.io/projected/3d77f97d-a54d-4e24-a128-e15fa735e656-kube-api-access-qj5dz\") pod \"e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.781784 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.781509 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:17.809405 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:17.809370 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:18.064828 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:18.064797 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv"] Oct 31 04:12:18.066486 ip-10-0-142-76 kubenswrapper[5490]: W1031 04:12:18.066450 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d77f97d_a54d_4e24_a128_e15fa735e656.slice/crio-a60c067dcf6fc26fb8d1753afac21081b39586bb8edde40f28c99598a2660615 WatchSource:0}: Error finding container a60c067dcf6fc26fb8d1753afac21081b39586bb8edde40f28c99598a2660615: Status 404 returned error can't find the container with id a60c067dcf6fc26fb8d1753afac21081b39586bb8edde40f28c99598a2660615 Oct 31 04:12:18.068852 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:18.068818 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:12:18.068967 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:18.068905 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:12:18.068967 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:18.068951 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:12:18.659569 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:18.659527 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" event={"ID":"3d77f97d-a54d-4e24-a128-e15fa735e656","Type":"ContainerStarted","Data":"84d0e42628c368d8c19038ca5afb1b77676bc55e9f3e7544996a5968ce1beca8"} Oct 31 04:12:18.660119 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:18.659615 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" event={"ID":"3d77f97d-a54d-4e24-a128-e15fa735e656","Type":"ContainerStarted","Data":"a60c067dcf6fc26fb8d1753afac21081b39586bb8edde40f28c99598a2660615"} Oct 31 04:12:18.682904 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:18.682849 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" podStartSLOduration=1.682832609 podStartE2EDuration="1.682832609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:18.680159172 +0000 UTC m=+3373.214854127" watchObservedRunningTime="2026-10-31 04:12:18.682832609 +0000 UTC m=+3373.217527565" Oct 31 04:12:18.781624 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:18.781586 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:19.115400 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:19.115307 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:19.238011 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:19.237971 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:19.787726 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:19.787669 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:20.189407 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:20.189364 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:20.330359 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:20.330311 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:20.671304 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:20.671274 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:20.672418 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:20.672398 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:21.313099 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.313062 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k"] Oct 31 04:12:21.317040 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.317019 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" Oct 31 04:12:21.326253 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.326225 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4bd78aa5-serving-cert\"" Oct 31 04:12:21.327471 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.326943 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k"] Oct 31 04:12:21.407517 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.407485 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:21.459686 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.459642 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fad1939-9f82-4594-917a-8c2ea8f771cb-maas-api-tls\") pod \"maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k\" (UID: \"5fad1939-9f82-4594-917a-8c2ea8f771cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" Oct 31 04:12:21.459867 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.459755 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fad1939-9f82-4594-917a-8c2ea8f771cb-openshift-service-ca\") pod \"maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k\" (UID: \"5fad1939-9f82-4594-917a-8c2ea8f771cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" Oct 31 04:12:21.459867 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.459786 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jchs\" (UniqueName: \"kubernetes.io/projected/5fad1939-9f82-4594-917a-8c2ea8f771cb-kube-api-access-6jchs\") pod \"maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k\" (UID: \"5fad1939-9f82-4594-917a-8c2ea8f771cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" Oct 31 04:12:21.459967 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.459876 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fad1939-9f82-4594-917a-8c2ea8f771cb-metrics-certs\") pod \"maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k\" (UID: \"5fad1939-9f82-4594-917a-8c2ea8f771cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" Oct 31 04:12:21.548577 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.548528 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:21.560934 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.560904 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fad1939-9f82-4594-917a-8c2ea8f771cb-maas-api-tls\") pod \"maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k\" (UID: \"5fad1939-9f82-4594-917a-8c2ea8f771cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" Oct 31 04:12:21.561111 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.560957 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fad1939-9f82-4594-917a-8c2ea8f771cb-openshift-service-ca\") pod \"maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k\" (UID: \"5fad1939-9f82-4594-917a-8c2ea8f771cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" Oct 31 04:12:21.561111 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.560992 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jchs\" (UniqueName: \"kubernetes.io/projected/5fad1939-9f82-4594-917a-8c2ea8f771cb-kube-api-access-6jchs\") pod \"maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k\" (UID: \"5fad1939-9f82-4594-917a-8c2ea8f771cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" Oct 31 04:12:21.561111 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.561023 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fad1939-9f82-4594-917a-8c2ea8f771cb-metrics-certs\") pod \"maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k\" (UID: \"5fad1939-9f82-4594-917a-8c2ea8f771cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" Oct 31 04:12:21.561778 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.561753 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fad1939-9f82-4594-917a-8c2ea8f771cb-openshift-service-ca\") pod \"maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k\" (UID: \"5fad1939-9f82-4594-917a-8c2ea8f771cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" Oct 31 04:12:21.564939 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.564877 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fad1939-9f82-4594-917a-8c2ea8f771cb-metrics-certs\") pod \"maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k\" (UID: \"5fad1939-9f82-4594-917a-8c2ea8f771cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" Oct 31 04:12:21.565550 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.565519 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fad1939-9f82-4594-917a-8c2ea8f771cb-maas-api-tls\") pod \"maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k\" (UID: \"5fad1939-9f82-4594-917a-8c2ea8f771cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" Oct 31 04:12:21.595721 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.578382 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jchs\" (UniqueName: \"kubernetes.io/projected/5fad1939-9f82-4594-917a-8c2ea8f771cb-kube-api-access-6jchs\") pod \"maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k\" (UID: \"5fad1939-9f82-4594-917a-8c2ea8f771cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" Oct 31 04:12:21.647912 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.647872 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" Oct 31 04:12:21.688722 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.688670 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:21.945979 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:21.945947 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k"] Oct 31 04:12:21.947241 ip-10-0-142-76 kubenswrapper[5490]: W1031 04:12:21.947214 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fad1939_9f82_4594_917a_8c2ea8f771cb.slice/crio-8d041a1e1ed4e66755a33254a51bc65017d10bc51bc8223e00c1f16a344b6aa0 WatchSource:0}: Error finding container 8d041a1e1ed4e66755a33254a51bc65017d10bc51bc8223e00c1f16a344b6aa0: Status 404 returned error can't find the container with id 8d041a1e1ed4e66755a33254a51bc65017d10bc51bc8223e00c1f16a344b6aa0 Oct 31 04:12:22.209430 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:22.209334 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:22.691952 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:22.691906 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" event={"ID":"5fad1939-9f82-4594-917a-8c2ea8f771cb","Type":"ContainerStarted","Data":"8e9f6d573684be7e4adb18a725fa2ea49e4cc669aa6186c34e027668e0c19a7f"} Oct 31 04:12:22.692362 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:22.692007 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" Oct 31 04:12:22.692362 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:22.692032 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" event={"ID":"5fad1939-9f82-4594-917a-8c2ea8f771cb","Type":"ContainerStarted","Data":"8d041a1e1ed4e66755a33254a51bc65017d10bc51bc8223e00c1f16a344b6aa0"} Oct 31 04:12:22.894133 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:22.894090 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:23.520468 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:23.520434 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:24.399234 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:24.399193 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:25.035271 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:25.035230 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:25.256264 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:25.256208 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" podStartSLOduration=3.857103996 podStartE2EDuration="4.256187821s" totalImagesPullingTime="399.083825ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:22.708904514 +0000 UTC m=+3377.243599470" watchObservedRunningTime="2026-10-31 04:12:25.256187821 +0000 UTC m=+3379.790882985" Oct 31 04:12:25.259154 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:25.259120 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh"] Oct 31 04:12:25.259333 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:25.259314 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" podUID="6f4d14ca-ec36-4daa-8971-93c7dd709b48" containerName="maas-api" containerID="cri-o://b012e17c9dc41fe2d34e8a61df9f0477d53fac64981e794e80574219cd79ecc7" gracePeriod=30 Oct 31 04:12:25.716393 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:25.716354 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:26.417549 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.417490 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:26.562482 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.562455 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" Oct 31 04:12:26.714456 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.714423 5490 generic.go:356] "Generic (PLEG): container finished" podID="6f4d14ca-ec36-4daa-8971-93c7dd709b48" containerID="b012e17c9dc41fe2d34e8a61df9f0477d53fac64981e794e80574219cd79ecc7" exitCode=0 Oct 31 04:12:26.714648 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.714471 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" Oct 31 04:12:26.714648 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.714508 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" event={"ID":"6f4d14ca-ec36-4daa-8971-93c7dd709b48","Type":"ContainerDied","Data":"b012e17c9dc41fe2d34e8a61df9f0477d53fac64981e794e80574219cd79ecc7"} Oct 31 04:12:26.714648 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.714566 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh" event={"ID":"6f4d14ca-ec36-4daa-8971-93c7dd709b48","Type":"ContainerDied","Data":"89cfcf2928ace145b676860f6394b9fae824c4cc7e87e514d6a478382d8f3fd7"} Oct 31 04:12:26.714648 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.714596 5490 scope.go:122] "RemoveContainer" containerID="b012e17c9dc41fe2d34e8a61df9f0477d53fac64981e794e80574219cd79ecc7" Oct 31 04:12:26.718808 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.718785 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f4d14ca-ec36-4daa-8971-93c7dd709b48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f4d14ca-ec36-4daa-8971-93c7dd709b48-metrics-certs\") pod \"6f4d14ca-ec36-4daa-8971-93c7dd709b48\" (UID: \"6f4d14ca-ec36-4daa-8971-93c7dd709b48\") " Oct 31 04:12:26.719239 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.718815 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f4d14ca-ec36-4daa-8971-93c7dd709b48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4d14ca-ec36-4daa-8971-93c7dd709b48-openshift-service-ca\") pod \"6f4d14ca-ec36-4daa-8971-93c7dd709b48\" (UID: \"6f4d14ca-ec36-4daa-8971-93c7dd709b48\") " Oct 31 04:12:26.719239 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.719031 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4d14ca-ec36-4daa-8971-93c7dd709b48-kube-api-access-mntm7\" (UniqueName: \"kubernetes.io/projected/6f4d14ca-ec36-4daa-8971-93c7dd709b48-kube-api-access-mntm7\") pod \"6f4d14ca-ec36-4daa-8971-93c7dd709b48\" (UID: \"6f4d14ca-ec36-4daa-8971-93c7dd709b48\") " Oct 31 04:12:26.719239 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.719087 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f4d14ca-ec36-4daa-8971-93c7dd709b48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4d14ca-ec36-4daa-8971-93c7dd709b48-maas-api-tls\") pod \"6f4d14ca-ec36-4daa-8971-93c7dd709b48\" (UID: \"6f4d14ca-ec36-4daa-8971-93c7dd709b48\") " Oct 31 04:12:26.719771 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.719718 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4d14ca-ec36-4daa-8971-93c7dd709b48-openshift-service-ca" pod "6f4d14ca-ec36-4daa-8971-93c7dd709b48" (UID: "6f4d14ca-ec36-4daa-8971-93c7dd709b48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:12:26.721632 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.721591 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f4d14ca-ec36-4daa-8971-93c7dd709b48-metrics-certs" pod "6f4d14ca-ec36-4daa-8971-93c7dd709b48" (UID: "6f4d14ca-ec36-4daa-8971-93c7dd709b48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:12:26.721767 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.721657 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4d14ca-ec36-4daa-8971-93c7dd709b48-kube-api-access-mntm7" pod "6f4d14ca-ec36-4daa-8971-93c7dd709b48" (UID: "6f4d14ca-ec36-4daa-8971-93c7dd709b48"). InnerVolumeSpecName "kube-api-access-mntm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:12:26.722115 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.722090 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f4d14ca-ec36-4daa-8971-93c7dd709b48-maas-api-tls" pod "6f4d14ca-ec36-4daa-8971-93c7dd709b48" (UID: "6f4d14ca-ec36-4daa-8971-93c7dd709b48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:12:26.748660 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.747856 5490 scope.go:122] "RemoveContainer" containerID="b012e17c9dc41fe2d34e8a61df9f0477d53fac64981e794e80574219cd79ecc7" Oct 31 04:12:26.748660 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:12:26.748351 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b012e17c9dc41fe2d34e8a61df9f0477d53fac64981e794e80574219cd79ecc7\": container with ID starting with b012e17c9dc41fe2d34e8a61df9f0477d53fac64981e794e80574219cd79ecc7 not found: ID does not exist" containerID="b012e17c9dc41fe2d34e8a61df9f0477d53fac64981e794e80574219cd79ecc7" Oct 31 04:12:26.748660 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.748390 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b012e17c9dc41fe2d34e8a61df9f0477d53fac64981e794e80574219cd79ecc7"} err="failed to get container status \"b012e17c9dc41fe2d34e8a61df9f0477d53fac64981e794e80574219cd79ecc7\": rpc error: code = NotFound desc = could not find container \"b012e17c9dc41fe2d34e8a61df9f0477d53fac64981e794e80574219cd79ecc7\": container with ID starting with b012e17c9dc41fe2d34e8a61df9f0477d53fac64981e794e80574219cd79ecc7 not found: ID does not exist" Oct 31 04:12:26.820987 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.820948 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f4d14ca-ec36-4daa-8971-93c7dd709b48-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:26.820987 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.820987 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f4d14ca-ec36-4daa-8971-93c7dd709b48-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:26.821217 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.820999 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mntm7\" (UniqueName: \"kubernetes.io/projected/6f4d14ca-ec36-4daa-8971-93c7dd709b48-kube-api-access-mntm7\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:26.821217 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:26.821009 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f4d14ca-ec36-4daa-8971-93c7dd709b48-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:27.052033 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:27.051956 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh"] Oct 31 04:12:27.055603 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:27.055567 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-78fdcdcc4-gp5mh"] Oct 31 04:12:27.098052 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:27.098013 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:27.574105 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:27.574067 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f4d14ca-ec36-4daa-8971-93c7dd709b48" path="/var/lib/kubelet/pods/6f4d14ca-ec36-4daa-8971-93c7dd709b48/volumes" Oct 31 04:12:27.800984 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:27.800948 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:28.503234 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:28.503192 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:28.703364 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:28.703328 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" Oct 31 04:12:29.294526 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:29.294485 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:29.999999 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:29.999951 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:30.687620 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:30.687577 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:31.678425 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:31.678372 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:32.313854 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:32.309017 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:32.756272 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:32.756231 5490 generic.go:356] "Generic (PLEG): container finished" podID="3d77f97d-a54d-4e24-a128-e15fa735e656" containerID="84d0e42628c368d8c19038ca5afb1b77676bc55e9f3e7544996a5968ce1beca8" exitCode=0 Oct 31 04:12:32.756512 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:32.756309 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" event={"ID":"3d77f97d-a54d-4e24-a128-e15fa735e656","Type":"ContainerDied","Data":"84d0e42628c368d8c19038ca5afb1b77676bc55e9f3e7544996a5968ce1beca8"} Oct 31 04:12:32.756748 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:32.756732 5490 scope.go:122] "RemoveContainer" containerID="84d0e42628c368d8c19038ca5afb1b77676bc55e9f3e7544996a5968ce1beca8" Oct 31 04:12:32.758151 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:32.758125 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:12:32.758239 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:32.758194 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:12:32.758276 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:32.758238 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:12:32.985811 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:32.985756 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:33.745702 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:33.741765 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:33.764985 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:33.764946 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" event={"ID":"3d77f97d-a54d-4e24-a128-e15fa735e656","Type":"ContainerStarted","Data":"73c74a8ffbd2daa6649d02c3b5cf699af8fbad49eac8cb99d32e1f4bf635ec07"} Oct 31 04:12:33.782457 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:33.782428 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:33.783877 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:33.783849 5490 patch_prober.go:29] interesting pod/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 31 04:12:33.783999 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:33.783899 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" podUID="3d77f97d-a54d-4e24-a128-e15fa735e656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 31 04:12:34.450448 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:34.450410 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:34.782182 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:34.782090 5490 patch_prober.go:29] interesting pod/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 31 04:12:34.782182 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:34.782151 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" podUID="3d77f97d-a54d-4e24-a128-e15fa735e656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 31 04:12:35.220953 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:35.220910 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:35.671499 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:35.671456 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:35.782739 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:35.782702 5490 patch_prober.go:29] interesting pod/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 31 04:12:35.783168 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:35.782767 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" podUID="3d77f97d-a54d-4e24-a128-e15fa735e656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 31 04:12:35.920846 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:35.920787 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:36.641721 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:36.640213 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:36.782737 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:36.782667 5490 patch_prober.go:29] interesting pod/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 31 04:12:36.782957 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:36.782769 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" podUID="3d77f97d-a54d-4e24-a128-e15fa735e656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 31 04:12:37.316391 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:37.316347 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:37.782659 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:37.782616 5490 patch_prober.go:29] interesting pod/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 31 04:12:37.782893 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:37.782718 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" podUID="3d77f97d-a54d-4e24-a128-e15fa735e656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 31 04:12:38.008491 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:38.008455 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:38.715644 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:38.715593 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:38.846776 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:38.846740 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:38.848412 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:38.848380 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:39.405271 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:39.405216 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:40.093212 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:40.093172 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:40.812314 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:40.812283 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:41.033020 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.032975 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:41.165570 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.165264 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:41.233150 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.233112 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k"] Oct 31 04:12:41.233426 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.233394 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" podUID="5fad1939-9f82-4594-917a-8c2ea8f771cb" containerName="maas-api" containerID="cri-o://8e9f6d573684be7e4adb18a725fa2ea49e4cc669aa6186c34e027668e0c19a7f" gracePeriod=30 Oct 31 04:12:41.317500 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.317446 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:41.453534 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.453421 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:41.528290 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.528256 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" Oct 31 04:12:41.591983 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.591947 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:41.614565 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.614527 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fad1939-9f82-4594-917a-8c2ea8f771cb-kube-api-access-6jchs\" (UniqueName: \"kubernetes.io/projected/5fad1939-9f82-4594-917a-8c2ea8f771cb-kube-api-access-6jchs\") pod \"5fad1939-9f82-4594-917a-8c2ea8f771cb\" (UID: \"5fad1939-9f82-4594-917a-8c2ea8f771cb\") " Oct 31 04:12:41.614774 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.614605 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fad1939-9f82-4594-917a-8c2ea8f771cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fad1939-9f82-4594-917a-8c2ea8f771cb-maas-api-tls\") pod \"5fad1939-9f82-4594-917a-8c2ea8f771cb\" (UID: \"5fad1939-9f82-4594-917a-8c2ea8f771cb\") " Oct 31 04:12:41.614774 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.614749 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fad1939-9f82-4594-917a-8c2ea8f771cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fad1939-9f82-4594-917a-8c2ea8f771cb-metrics-certs\") pod \"5fad1939-9f82-4594-917a-8c2ea8f771cb\" (UID: \"5fad1939-9f82-4594-917a-8c2ea8f771cb\") " Oct 31 04:12:41.614871 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.614824 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fad1939-9f82-4594-917a-8c2ea8f771cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fad1939-9f82-4594-917a-8c2ea8f771cb-openshift-service-ca\") pod \"5fad1939-9f82-4594-917a-8c2ea8f771cb\" (UID: \"5fad1939-9f82-4594-917a-8c2ea8f771cb\") " Oct 31 04:12:41.615225 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.615194 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fad1939-9f82-4594-917a-8c2ea8f771cb-openshift-service-ca" pod "5fad1939-9f82-4594-917a-8c2ea8f771cb" (UID: "5fad1939-9f82-4594-917a-8c2ea8f771cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:12:41.616897 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.616868 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fad1939-9f82-4594-917a-8c2ea8f771cb-metrics-certs" pod "5fad1939-9f82-4594-917a-8c2ea8f771cb" (UID: "5fad1939-9f82-4594-917a-8c2ea8f771cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:12:41.617186 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.617163 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fad1939-9f82-4594-917a-8c2ea8f771cb-kube-api-access-6jchs" pod "5fad1939-9f82-4594-917a-8c2ea8f771cb" (UID: "5fad1939-9f82-4594-917a-8c2ea8f771cb"). InnerVolumeSpecName "kube-api-access-6jchs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:12:41.617350 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.617329 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fad1939-9f82-4594-917a-8c2ea8f771cb-maas-api-tls" pod "5fad1939-9f82-4594-917a-8c2ea8f771cb" (UID: "5fad1939-9f82-4594-917a-8c2ea8f771cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:12:41.716253 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.716164 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fad1939-9f82-4594-917a-8c2ea8f771cb-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:41.716253 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.716200 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fad1939-9f82-4594-917a-8c2ea8f771cb-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:41.716253 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.716212 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jchs\" (UniqueName: \"kubernetes.io/projected/5fad1939-9f82-4594-917a-8c2ea8f771cb-kube-api-access-6jchs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:41.716253 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.716223 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fad1939-9f82-4594-917a-8c2ea8f771cb-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:41.814626 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.814586 5490 generic.go:356] "Generic (PLEG): container finished" podID="5fad1939-9f82-4594-917a-8c2ea8f771cb" containerID="8e9f6d573684be7e4adb18a725fa2ea49e4cc669aa6186c34e027668e0c19a7f" exitCode=0 Oct 31 04:12:41.815099 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.814641 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" Oct 31 04:12:41.815099 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.814658 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" event={"ID":"5fad1939-9f82-4594-917a-8c2ea8f771cb","Type":"ContainerDied","Data":"8e9f6d573684be7e4adb18a725fa2ea49e4cc669aa6186c34e027668e0c19a7f"} Oct 31 04:12:41.815099 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.814733 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k" event={"ID":"5fad1939-9f82-4594-917a-8c2ea8f771cb","Type":"ContainerDied","Data":"8d041a1e1ed4e66755a33254a51bc65017d10bc51bc8223e00c1f16a344b6aa0"} Oct 31 04:12:41.815099 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.814757 5490 scope.go:122] "RemoveContainer" containerID="8e9f6d573684be7e4adb18a725fa2ea49e4cc669aa6186c34e027668e0c19a7f" Oct 31 04:12:41.825585 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.825566 5490 scope.go:122] "RemoveContainer" containerID="8e9f6d573684be7e4adb18a725fa2ea49e4cc669aa6186c34e027668e0c19a7f" Oct 31 04:12:41.825955 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:12:41.825929 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e9f6d573684be7e4adb18a725fa2ea49e4cc669aa6186c34e027668e0c19a7f\": container with ID starting with 8e9f6d573684be7e4adb18a725fa2ea49e4cc669aa6186c34e027668e0c19a7f not found: ID does not exist" containerID="8e9f6d573684be7e4adb18a725fa2ea49e4cc669aa6186c34e027668e0c19a7f" Oct 31 04:12:41.826004 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.825970 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e9f6d573684be7e4adb18a725fa2ea49e4cc669aa6186c34e027668e0c19a7f"} err="failed to get container status \"8e9f6d573684be7e4adb18a725fa2ea49e4cc669aa6186c34e027668e0c19a7f\": rpc error: code = NotFound desc = could not find container \"8e9f6d573684be7e4adb18a725fa2ea49e4cc669aa6186c34e027668e0c19a7f\": container with ID starting with 8e9f6d573684be7e4adb18a725fa2ea49e4cc669aa6186c34e027668e0c19a7f not found: ID does not exist" Oct 31 04:12:41.839422 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.839390 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k"] Oct 31 04:12:41.841854 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:41.841829 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4bd78aa5-6b85d9df55-v2q7k"] Oct 31 04:12:43.574452 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:43.574413 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fad1939-9f82-4594-917a-8c2ea8f771cb" path="/var/lib/kubelet/pods/5fad1939-9f82-4594-917a-8c2ea8f771cb/volumes" Oct 31 04:12:47.088977 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:47.088932 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv"] Oct 31 04:12:47.089707 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:47.089217 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" podUID="3d77f97d-a54d-4e24-a128-e15fa735e656" containerName="istio-proxy" containerID="cri-o://73c74a8ffbd2daa6649d02c3b5cf699af8fbad49eac8cb99d32e1f4bf635ec07" gracePeriod=30 Oct 31 04:12:47.131912 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:12:47.131867 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:12:47.175099 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:47.175067 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:48.342992 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.342950 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:48.658715 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.658654 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424"] Oct 31 04:12:48.659194 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.659177 5490 cpu_manager.go:414] "containerMap: removing container" podUID="6f4d14ca-ec36-4daa-8971-93c7dd709b48" containerName="maas-api" Oct 31 04:12:48.659273 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.659195 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4d14ca-ec36-4daa-8971-93c7dd709b48" containerName="maas-api" Oct 31 04:12:48.659273 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.659228 5490 cpu_manager.go:414] "containerMap: removing container" podUID="5fad1939-9f82-4594-917a-8c2ea8f771cb" containerName="maas-api" Oct 31 04:12:48.659273 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.659233 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fad1939-9f82-4594-917a-8c2ea8f771cb" containerName="maas-api" Oct 31 04:12:48.659443 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.659341 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4d14ca-ec36-4daa-8971-93c7dd709b48" containerName="maas-api" Oct 31 04:12:48.659443 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.659359 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fad1939-9f82-4594-917a-8c2ea8f771cb" containerName="maas-api" Oct 31 04:12:48.664199 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.664172 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.677734 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.677659 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-5e424f-openshift-default-dockercfg-64qmw\"" Oct 31 04:12:48.679992 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.679959 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424"] Oct 31 04:12:48.750273 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.750234 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:48.788536 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.788498 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsm6r\" (UniqueName: \"kubernetes.io/projected/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-kube-api-access-lsm6r\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.788754 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.788567 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-podinfo\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.788754 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.788664 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-token\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.788872 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.788768 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-data\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.788872 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.788796 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-workload-certs\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.788965 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.788868 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-envoy\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.788965 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.788897 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-credential-socket\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.788965 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.788921 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istiod-ca-cert\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.789108 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.788989 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-workload-socket\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.890412 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.890371 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-envoy\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.890412 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.890414 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-credential-socket\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.890717 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.890438 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istiod-ca-cert\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.890717 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.890475 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-workload-socket\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.890717 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.890516 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsm6r\" (UniqueName: \"kubernetes.io/projected/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-kube-api-access-lsm6r\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.890717 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.890566 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-podinfo\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.890717 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.890594 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-token\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.890717 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.890650 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-data\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.890717 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.890688 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-workload-certs\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.891129 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.890882 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-workload-socket\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.891129 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.890900 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-credential-socket\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.891129 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.891052 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-workload-certs\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.891307 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.891124 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-data\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.891307 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.891274 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istiod-ca-cert\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.893722 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.893253 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-envoy\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.893722 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.893670 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-podinfo\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.898768 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.898731 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-token\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.898892 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.898812 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsm6r\" (UniqueName: \"kubernetes.io/projected/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-kube-api-access-lsm6r\") pod \"e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:48.953443 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.953324 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:48.979811 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:48.979776 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:49.265255 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:49.265222 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424"] Oct 31 04:12:49.266949 ip-10-0-142-76 kubenswrapper[5490]: W1031 04:12:49.266915 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6229f3_9eaa_4d9e_ac22_be70a9060eca.slice/crio-0e6393c6e5ce04de15e9211502785bbe907788cfb5bb287af6d6444c44bcfe84 WatchSource:0}: Error finding container 0e6393c6e5ce04de15e9211502785bbe907788cfb5bb287af6d6444c44bcfe84: Status 404 returned error can't find the container with id 0e6393c6e5ce04de15e9211502785bbe907788cfb5bb287af6d6444c44bcfe84 Oct 31 04:12:49.269357 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:49.269326 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:12:49.269438 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:49.269391 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:12:49.269438 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:49.269424 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:12:49.865291 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:49.865254 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" event={"ID":"ba6229f3-9eaa-4d9e-ac22-be70a9060eca","Type":"ContainerStarted","Data":"252a43e73e28507eb9b5f19e9849ee4f2743776ff6dfe1043761ed2f5f29859e"} Oct 31 04:12:49.865782 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:49.865309 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" event={"ID":"ba6229f3-9eaa-4d9e-ac22-be70a9060eca","Type":"ContainerStarted","Data":"0e6393c6e5ce04de15e9211502785bbe907788cfb5bb287af6d6444c44bcfe84"} Oct 31 04:12:49.885010 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:49.884953 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" podStartSLOduration=1.884937994 podStartE2EDuration="1.884937994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:49.883184578 +0000 UTC m=+3404.417879533" watchObservedRunningTime="2026-10-31 04:12:49.884937994 +0000 UTC m=+3404.419632949" Oct 31 04:12:49.980790 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:49.980740 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:50.986438 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:50.986398 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:51.876393 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:51.876361 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:51.877452 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:51.877428 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:12:52.066612 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.066572 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:52.201114 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.201079 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:52.356020 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.355992 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:52.431255 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.431209 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d77f97d-a54d-4e24-a128-e15fa735e656-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d77f97d-a54d-4e24-a128-e15fa735e656-istiod-ca-cert\") pod \"3d77f97d-a54d-4e24-a128-e15fa735e656\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " Oct 31 04:12:52.431473 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.431295 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-workload-certs\") pod \"3d77f97d-a54d-4e24-a128-e15fa735e656\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " Oct 31 04:12:52.431473 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.431319 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-token\" (UniqueName: \"kubernetes.io/projected/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-token\") pod \"3d77f97d-a54d-4e24-a128-e15fa735e656\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " Oct 31 04:12:52.431473 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.431344 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d77f97d-a54d-4e24-a128-e15fa735e656-kube-api-access-qj5dz\" (UniqueName: \"kubernetes.io/projected/3d77f97d-a54d-4e24-a128-e15fa735e656-kube-api-access-qj5dz\") pod \"3d77f97d-a54d-4e24-a128-e15fa735e656\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " Oct 31 04:12:52.431473 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.431386 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-credential-socket\") pod \"3d77f97d-a54d-4e24-a128-e15fa735e656\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " Oct 31 04:12:52.431473 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.431436 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-podinfo\") pod \"3d77f97d-a54d-4e24-a128-e15fa735e656\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " Oct 31 04:12:52.431793 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.431497 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-workload-socket\") pod \"3d77f97d-a54d-4e24-a128-e15fa735e656\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " Oct 31 04:12:52.431793 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.431563 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-envoy\") pod \"3d77f97d-a54d-4e24-a128-e15fa735e656\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " Oct 31 04:12:52.431793 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.431617 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-data\") pod \"3d77f97d-a54d-4e24-a128-e15fa735e656\" (UID: \"3d77f97d-a54d-4e24-a128-e15fa735e656\") " Oct 31 04:12:52.431793 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.431712 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-workload-certs" pod "3d77f97d-a54d-4e24-a128-e15fa735e656" (UID: "3d77f97d-a54d-4e24-a128-e15fa735e656"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:12:52.431793 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.431769 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d77f97d-a54d-4e24-a128-e15fa735e656-istiod-ca-cert" pod "3d77f97d-a54d-4e24-a128-e15fa735e656" (UID: "3d77f97d-a54d-4e24-a128-e15fa735e656"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:12:52.431793 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.431774 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-credential-socket" pod "3d77f97d-a54d-4e24-a128-e15fa735e656" (UID: "3d77f97d-a54d-4e24-a128-e15fa735e656"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:12:52.432176 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.431905 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-workload-socket" pod "3d77f97d-a54d-4e24-a128-e15fa735e656" (UID: "3d77f97d-a54d-4e24-a128-e15fa735e656"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:12:52.432176 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.431990 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-data" pod "3d77f97d-a54d-4e24-a128-e15fa735e656" (UID: "3d77f97d-a54d-4e24-a128-e15fa735e656"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:12:52.432269 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.432195 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:52.432269 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.432215 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d77f97d-a54d-4e24-a128-e15fa735e656-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:52.432269 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.432229 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:52.432269 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.432242 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:52.432269 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.432251 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:52.434260 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.434235 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-envoy" pod "3d77f97d-a54d-4e24-a128-e15fa735e656" (UID: "3d77f97d-a54d-4e24-a128-e15fa735e656"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:12:52.434354 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.434234 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d77f97d-a54d-4e24-a128-e15fa735e656-kube-api-access-qj5dz" pod "3d77f97d-a54d-4e24-a128-e15fa735e656" (UID: "3d77f97d-a54d-4e24-a128-e15fa735e656"). InnerVolumeSpecName "kube-api-access-qj5dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:12:52.434354 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.434282 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-podinfo" pod "3d77f97d-a54d-4e24-a128-e15fa735e656" (UID: "3d77f97d-a54d-4e24-a128-e15fa735e656"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:12:52.434354 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.434289 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-token" pod "3d77f97d-a54d-4e24-a128-e15fa735e656" (UID: "3d77f97d-a54d-4e24-a128-e15fa735e656"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:12:52.533202 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.533109 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:52.533202 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.533147 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj5dz\" (UniqueName: \"kubernetes.io/projected/3d77f97d-a54d-4e24-a128-e15fa735e656-kube-api-access-qj5dz\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:52.533202 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.533158 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:52.533202 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.533167 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d77f97d-a54d-4e24-a128-e15fa735e656-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:12:52.884236 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.884133 5490 generic.go:356] "Generic (PLEG): container finished" podID="3d77f97d-a54d-4e24-a128-e15fa735e656" containerID="73c74a8ffbd2daa6649d02c3b5cf699af8fbad49eac8cb99d32e1f4bf635ec07" exitCode=0 Oct 31 04:12:52.884236 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.884197 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" Oct 31 04:12:52.884236 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.884218 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" event={"ID":"3d77f97d-a54d-4e24-a128-e15fa735e656","Type":"ContainerDied","Data":"73c74a8ffbd2daa6649d02c3b5cf699af8fbad49eac8cb99d32e1f4bf635ec07"} Oct 31 04:12:52.884500 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.884281 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv" event={"ID":"3d77f97d-a54d-4e24-a128-e15fa735e656","Type":"ContainerDied","Data":"a60c067dcf6fc26fb8d1753afac21081b39586bb8edde40f28c99598a2660615"} Oct 31 04:12:52.884500 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.884308 5490 scope.go:122] "RemoveContainer" containerID="73c74a8ffbd2daa6649d02c3b5cf699af8fbad49eac8cb99d32e1f4bf635ec07" Oct 31 04:12:52.896911 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.896882 5490 scope.go:122] "RemoveContainer" containerID="84d0e42628c368d8c19038ca5afb1b77676bc55e9f3e7544996a5968ce1beca8" Oct 31 04:12:52.908008 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.907584 5490 scope.go:122] "RemoveContainer" containerID="73c74a8ffbd2daa6649d02c3b5cf699af8fbad49eac8cb99d32e1f4bf635ec07" Oct 31 04:12:52.909144 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.909121 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv"] Oct 31 04:12:52.910317 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:12:52.910291 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73c74a8ffbd2daa6649d02c3b5cf699af8fbad49eac8cb99d32e1f4bf635ec07\": container with ID starting with 73c74a8ffbd2daa6649d02c3b5cf699af8fbad49eac8cb99d32e1f4bf635ec07 not found: ID does not exist" containerID="73c74a8ffbd2daa6649d02c3b5cf699af8fbad49eac8cb99d32e1f4bf635ec07" Oct 31 04:12:52.910395 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.910331 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c74a8ffbd2daa6649d02c3b5cf699af8fbad49eac8cb99d32e1f4bf635ec07"} err="failed to get container status \"73c74a8ffbd2daa6649d02c3b5cf699af8fbad49eac8cb99d32e1f4bf635ec07\": rpc error: code = NotFound desc = could not find container \"73c74a8ffbd2daa6649d02c3b5cf699af8fbad49eac8cb99d32e1f4bf635ec07\": container with ID starting with 73c74a8ffbd2daa6649d02c3b5cf699af8fbad49eac8cb99d32e1f4bf635ec07 not found: ID does not exist" Oct 31 04:12:52.910395 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.910359 5490 scope.go:122] "RemoveContainer" containerID="84d0e42628c368d8c19038ca5afb1b77676bc55e9f3e7544996a5968ce1beca8" Oct 31 04:12:52.910714 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:12:52.910693 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d0e42628c368d8c19038ca5afb1b77676bc55e9f3e7544996a5968ce1beca8\": container with ID starting with 84d0e42628c368d8c19038ca5afb1b77676bc55e9f3e7544996a5968ce1beca8 not found: ID does not exist" containerID="84d0e42628c368d8c19038ca5afb1b77676bc55e9f3e7544996a5968ce1beca8" Oct 31 04:12:52.910755 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.910722 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d0e42628c368d8c19038ca5afb1b77676bc55e9f3e7544996a5968ce1beca8"} err="failed to get container status \"84d0e42628c368d8c19038ca5afb1b77676bc55e9f3e7544996a5968ce1beca8\": rpc error: code = NotFound desc = could not find container \"84d0e42628c368d8c19038ca5afb1b77676bc55e9f3e7544996a5968ce1beca8\": container with ID starting with 84d0e42628c368d8c19038ca5afb1b77676bc55e9f3e7544996a5968ce1beca8 not found: ID does not exist" Oct 31 04:12:52.912168 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:52.912150 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4bd78aa5-openshift-default-86497dc789-t4xmv"] Oct 31 04:12:53.210858 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.210782 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc"] Oct 31 04:12:53.211635 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.211606 5490 cpu_manager.go:414] "containerMap: removing container" podUID="3d77f97d-a54d-4e24-a128-e15fa735e656" containerName="istio-proxy" Oct 31 04:12:53.211751 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.211638 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d77f97d-a54d-4e24-a128-e15fa735e656" containerName="istio-proxy" Oct 31 04:12:53.211819 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.211802 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d77f97d-a54d-4e24-a128-e15fa735e656" containerName="istio-proxy" Oct 31 04:12:53.217285 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.217248 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" Oct 31 04:12:53.220898 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.220873 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-5e424f-serving-cert\"" Oct 31 04:12:53.224825 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.224768 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc"] Oct 31 04:12:53.264334 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.264297 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:53.341017 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.340972 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xflvl\" (UniqueName: \"kubernetes.io/projected/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-kube-api-access-xflvl\") pod \"maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc\" (UID: \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" Oct 31 04:12:53.341219 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.341039 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-metrics-certs\") pod \"maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc\" (UID: \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" Oct 31 04:12:53.341219 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.341184 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc\" (UID: \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" Oct 31 04:12:53.341320 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.341241 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc\" (UID: \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" Oct 31 04:12:53.419004 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.418962 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:53.442903 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.442865 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-metrics-certs\") pod \"maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc\" (UID: \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" Oct 31 04:12:53.443098 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.442936 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc\" (UID: \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" Oct 31 04:12:53.443098 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.442974 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc\" (UID: \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" Oct 31 04:12:53.443098 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:12:53.443061 5490 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-serving-cert: secret "maas-api-e2e-authcfg-http-5e424f-serving-cert" not found Oct 31 04:12:53.443230 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.443099 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xflvl\" (UniqueName: \"kubernetes.io/projected/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-kube-api-access-xflvl\") pod \"maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc\" (UID: \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" Oct 31 04:12:53.443230 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:12:53.443132 5490 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-maas-api-tls podName:7e70b0f0-c12d-4d2d-9a61-e059c63c071d nodeName:}" failed. No retries permitted until 2026-10-31 04:12:53.943112238 +0000 UTC m=+3408.477807190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-maas-api-tls") pod "maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" (UID: "7e70b0f0-c12d-4d2d-9a61-e059c63c071d") : secret "maas-api-e2e-authcfg-http-5e424f-serving-cert" not found Oct 31 04:12:53.443801 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.443773 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc\" (UID: \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" Oct 31 04:12:53.445845 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.445823 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-metrics-certs\") pod \"maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc\" (UID: \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" Oct 31 04:12:53.452298 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.452246 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xflvl\" (UniqueName: \"kubernetes.io/projected/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-kube-api-access-xflvl\") pod \"maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc\" (UID: \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" Oct 31 04:12:53.564197 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.564102 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:53.574344 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.574311 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d77f97d-a54d-4e24-a128-e15fa735e656" path="/var/lib/kubelet/pods/3d77f97d-a54d-4e24-a128-e15fa735e656/volumes" Oct 31 04:12:53.948537 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.948496 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc\" (UID: \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" Oct 31 04:12:53.951030 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:53.951004 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc\" (UID: \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" Oct 31 04:12:54.081292 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:54.081250 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:54.141246 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:54.141195 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" Oct 31 04:12:54.428720 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:54.428652 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc"] Oct 31 04:12:54.430700 ip-10-0-142-76 kubenswrapper[5490]: W1031 04:12:54.430659 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e70b0f0_c12d_4d2d_9a61_e059c63c071d.slice/crio-359fc24f739ec3ea5a9fa462908b8adbc0aff230f46548c8f3037ad42c5feeb9 WatchSource:0}: Error finding container 359fc24f739ec3ea5a9fa462908b8adbc0aff230f46548c8f3037ad42c5feeb9: Status 404 returned error can't find the container with id 359fc24f739ec3ea5a9fa462908b8adbc0aff230f46548c8f3037ad42c5feeb9 Oct 31 04:12:54.848556 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:54.848511 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:54.898496 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:54.898452 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" event={"ID":"7e70b0f0-c12d-4d2d-9a61-e059c63c071d","Type":"ContainerStarted","Data":"359fc24f739ec3ea5a9fa462908b8adbc0aff230f46548c8f3037ad42c5feeb9"} Oct 31 04:12:55.495860 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:55.495809 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:55.922670 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:55.922629 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" event={"ID":"7e70b0f0-c12d-4d2d-9a61-e059c63c071d","Type":"ContainerStarted","Data":"fb1f26d5d694a4b8d9640c213fdf14bc272b85ed90ab8829c64abd0e15b7f79b"} Oct 31 04:12:55.923870 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:55.923813 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" Oct 31 04:12:55.946259 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:55.946200 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" podStartSLOduration=2.456491963 podStartE2EDuration="2.946184932s" totalImagesPullingTime="489.692969ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:55.941721407 +0000 UTC m=+3410.476416363" watchObservedRunningTime="2026-10-31 04:12:55.946184932 +0000 UTC m=+3410.480879886" Oct 31 04:12:56.324853 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:56.324754 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:57.016861 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:57.016819 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:57.711596 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:57.711550 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:58.415412 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:58.415364 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:59.182786 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:59.182750 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:12:59.879001 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:12:59.878955 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:00.608132 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:00.608092 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:01.312217 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:01.312175 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:01.957468 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:01.957430 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" Oct 31 04:13:02.018938 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:02.018892 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:02.906580 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:02.906531 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:03.611484 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:03.611426 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:04.288126 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:04.288084 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:04.911036 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:04.910994 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:05.518454 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:05.518407 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:06.002503 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:06.002467 5490 generic.go:356] "Generic (PLEG): container finished" podID="ba6229f3-9eaa-4d9e-ac22-be70a9060eca" containerID="252a43e73e28507eb9b5f19e9849ee4f2743776ff6dfe1043761ed2f5f29859e" exitCode=0 Oct 31 04:13:06.002749 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:06.002541 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" event={"ID":"ba6229f3-9eaa-4d9e-ac22-be70a9060eca","Type":"ContainerDied","Data":"252a43e73e28507eb9b5f19e9849ee4f2743776ff6dfe1043761ed2f5f29859e"} Oct 31 04:13:06.002989 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:06.002971 5490 scope.go:122] "RemoveContainer" containerID="252a43e73e28507eb9b5f19e9849ee4f2743776ff6dfe1043761ed2f5f29859e" Oct 31 04:13:06.004249 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:06.004215 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:13:06.004330 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:06.004315 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:13:06.004384 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:06.004355 5490 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 31 04:13:06.195465 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:06.195413 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:06.810185 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:06.810149 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:06.876328 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:06.876292 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:13:06.994774 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:06.994730 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:07.012046 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:07.012004 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" event={"ID":"ba6229f3-9eaa-4d9e-ac22-be70a9060eca","Type":"ContainerStarted","Data":"052b227570ab95e22bb84bbd89d478326bed9227401b782e6a3b5764fe8ac5d8"} Oct 31 04:13:07.522500 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:07.522459 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:07.980479 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:07.980435 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:13:07.981998 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:07.981975 5490 patch_prober.go:29] interesting pod/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 04:13:07.982144 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:07.982033 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" podUID="ba6229f3-9eaa-4d9e-ac22-be70a9060eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 04:13:08.266819 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:08.266723 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:08.980563 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:08.980525 5490 patch_prober.go:29] interesting pod/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 04:13:08.980996 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:08.980591 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" podUID="ba6229f3-9eaa-4d9e-ac22-be70a9060eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 04:13:09.063161 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:09.063116 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:09.779665 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:09.779628 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:09.980599 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:09.980558 5490 patch_prober.go:29] interesting pod/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 04:13:09.981180 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:09.980623 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" podUID="ba6229f3-9eaa-4d9e-ac22-be70a9060eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 04:13:10.355427 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:10.355391 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:10.484199 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:10.484161 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:10.621427 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:10.621318 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:10.980486 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:10.980443 5490 patch_prober.go:29] interesting pod/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 04:13:10.980758 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:10.980519 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" podUID="ba6229f3-9eaa-4d9e-ac22-be70a9060eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 04:13:11.314971 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:11.314874 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:11.980517 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:11.980478 5490 patch_prober.go:29] interesting pod/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 04:13:11.980745 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:11.980548 5490 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" podUID="ba6229f3-9eaa-4d9e-ac22-be70a9060eca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 04:13:12.146052 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:12.146003 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:12.458660 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:12.458615 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:12.801794 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:12.801708 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:12.984644 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:12.984615 5490 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:13:12.985103 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:12.985084 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:13:12.985873 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:12.985853 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:13:13.404362 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:13.404315 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:14.210920 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:14.210874 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:14.806249 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:14.806209 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:15.423688 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:15.423639 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:16.068702 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:16.068635 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:16.712481 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:16.712436 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:17.129373 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:13:17.129281 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:13:17.397421 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:17.397321 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:18.009242 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:18.009182 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:18.430159 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:18.430133 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ttmkb_400b676a-a3a0-459d-9da6-8d30b2d3b497/csi-driver/0.log" Oct 31 04:13:18.611633 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:18.611592 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:19.244042 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:19.244006 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:19.370097 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:19.370061 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:19.905508 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:19.905458 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:20.673531 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:20.673468 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:21.307628 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:21.307589 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:21.984942 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:21.984891 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:22.606472 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:22.606434 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:23.202176 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:23.202128 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:23.971640 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:23.971608 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:24.610730 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:24.610661 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:25.293033 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:25.292993 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:25.892239 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:25.892122 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:26.594704 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:26.594635 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:27.205658 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:27.205603 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:27.941468 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:27.941427 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:28.637260 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:28.634722 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:29.316264 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:29.316220 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:29.987984 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:29.987949 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:30.619662 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:30.619555 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:31.327447 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:31.327415 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:32.005232 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:32.004940 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:32.664445 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:32.664406 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:33.325478 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:33.325444 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:33.954134 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:33.954074 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:34.605143 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:34.605106 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:35.310171 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:35.310128 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:35.683523 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:35.683493 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ttmkb_400b676a-a3a0-459d-9da6-8d30b2d3b497/csi-driver/0.log" Oct 31 04:13:35.974474 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:35.974384 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:36.590934 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:36.590892 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:37.201313 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:37.201269 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:37.974740 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:37.974704 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:38.593376 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:38.593318 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:39.310845 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:39.310797 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:39.956963 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:39.956923 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:40.602116 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:40.602076 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:41.313860 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:41.313814 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:41.991388 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:41.991346 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:42.604981 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:42.604937 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:43.304947 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:43.304904 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:43.901784 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:43.901748 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:44.500374 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:44.500315 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:45.119396 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:45.119356 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:45.715267 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:45.715217 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:46.403555 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:46.403518 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:47.106053 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:47.105992 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:47.131543 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:13:47.131502 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:13:47.799614 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:47.799580 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:48.405770 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:48.405728 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:49.010408 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:49.010373 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:49.700211 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:49.700174 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:50.380160 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:50.380118 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:50.993413 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:50.993376 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:51.590343 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:51.590307 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:52.454629 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:52.454582 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:53.102297 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:53.102253 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:53.702467 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:53.702408 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:54.387133 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:54.387096 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:55.011415 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:55.011369 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:55.669913 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:55.669860 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:56.299505 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:56.299466 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:56.888728 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:56.888660 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:57.496422 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:57.496383 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:58.097791 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:58.097746 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:58.779325 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:58.779278 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:13:59.493370 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:13:59.493325 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:00.095918 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:00.095877 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:00.698899 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:00.698861 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:01.291251 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:01.291207 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:01.989438 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:01.989375 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:02.585315 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:02.585265 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:03.194928 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:03.194890 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:03.795448 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:03.795397 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:04.401122 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:04.401074 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:05.002067 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:05.002019 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:05.592750 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:05.592377 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:06.194809 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:06.194768 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:06.787538 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:06.787502 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:07.523206 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:07.523158 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:08.177796 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:08.177754 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:08.982792 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:08.982734 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:09.549060 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:09.549021 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:10.192391 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:10.192354 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:10.790483 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:10.790442 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:11.405829 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:11.405773 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:12.127793 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:12.127752 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:12.691231 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:12.691187 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:13.372591 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:13.372530 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:14.111535 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:14.111481 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:14.745645 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:14.745602 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:15.420757 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:15.420701 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:16.089059 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:16.089022 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:16.716698 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:16.716654 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:17.131487 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:14:17.131396 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:14:17.355599 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:17.355560 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:17.921074 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:17.920997 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:18.518562 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:18.518525 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:19.184311 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:19.184271 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:19.383171 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:19.383133 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:19.880557 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:19.880520 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:20.586756 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:20.586717 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:21.184624 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:21.184579 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:21.792269 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:21.792235 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:22.399274 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:22.399234 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:23.081022 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:23.080973 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:23.818780 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:23.818735 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:24.494701 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:24.494647 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:25.237727 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:25.237663 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:25.889005 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:25.888958 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:26.509624 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:26.509582 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:27.097593 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:27.097550 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:27.703589 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:27.703548 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:28.399929 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:28.399885 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:29.155068 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:29.155030 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:29.760913 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:29.760869 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:30.573216 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:30.573167 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:31.469958 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:31.469920 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:32.101627 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:32.101585 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:32.693221 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:32.693182 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:33.297326 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:33.297287 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:33.923717 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:33.920341 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:34.605066 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:34.605020 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:35.311957 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:35.311910 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:35.922540 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:35.922498 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:36.595523 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:36.595488 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:37.294425 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:37.294379 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:37.909839 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:37.908763 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:38.621093 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:38.621052 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:39.301473 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:39.301434 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:39.903965 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:39.903932 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:40.108897 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:40.108858 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:44.847186 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:44.847151 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc"] Oct 31 04:14:44.847755 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:44.847386 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" podUID="7e70b0f0-c12d-4d2d-9a61-e059c63c071d" containerName="maas-api" containerID="cri-o://fb1f26d5d694a4b8d9640c213fdf14bc272b85ed90ab8829c64abd0e15b7f79b" gracePeriod=30 Oct 31 04:14:44.945127 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:44.945089 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:45.102660 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.102625 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:45.116510 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.116482 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" Oct 31 04:14:45.236830 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.236787 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-metrics-certs\") pod \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\" (UID: \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\") " Oct 31 04:14:45.237056 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.236958 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-kube-api-access-xflvl\" (UniqueName: \"kubernetes.io/projected/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-kube-api-access-xflvl\") pod \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\" (UID: \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\") " Oct 31 04:14:45.237056 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.236992 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-maas-api-tls\") pod \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\" (UID: \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\") " Oct 31 04:14:45.237260 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.237091 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-openshift-service-ca\") pod \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\" (UID: \"7e70b0f0-c12d-4d2d-9a61-e059c63c071d\") " Oct 31 04:14:45.237575 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.237509 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-openshift-service-ca" pod "7e70b0f0-c12d-4d2d-9a61-e059c63c071d" (UID: "7e70b0f0-c12d-4d2d-9a61-e059c63c071d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:45.239508 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.239467 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-metrics-certs" pod "7e70b0f0-c12d-4d2d-9a61-e059c63c071d" (UID: "7e70b0f0-c12d-4d2d-9a61-e059c63c071d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:14:45.239787 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.239762 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-kube-api-access-xflvl" pod "7e70b0f0-c12d-4d2d-9a61-e059c63c071d" (UID: "7e70b0f0-c12d-4d2d-9a61-e059c63c071d"). InnerVolumeSpecName "kube-api-access-xflvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:45.239913 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.239834 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-maas-api-tls" pod "7e70b0f0-c12d-4d2d-9a61-e059c63c071d" (UID: "7e70b0f0-c12d-4d2d-9a61-e059c63c071d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:14:45.338949 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.338906 5490 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-metrics-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:14:45.338949 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.338949 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xflvl\" (UniqueName: \"kubernetes.io/projected/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-kube-api-access-xflvl\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:14:45.339185 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.338963 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-maas-api-tls\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:14:45.339185 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.338976 5490 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e70b0f0-c12d-4d2d-9a61-e059c63c071d-openshift-service-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:14:45.341118 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.341084 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:45.593041 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.593008 5490 generic.go:356] "Generic (PLEG): container finished" podID="7e70b0f0-c12d-4d2d-9a61-e059c63c071d" containerID="fb1f26d5d694a4b8d9640c213fdf14bc272b85ed90ab8829c64abd0e15b7f79b" exitCode=0 Oct 31 04:14:45.593203 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.593061 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" Oct 31 04:14:45.593203 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.593065 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" event={"ID":"7e70b0f0-c12d-4d2d-9a61-e059c63c071d","Type":"ContainerDied","Data":"fb1f26d5d694a4b8d9640c213fdf14bc272b85ed90ab8829c64abd0e15b7f79b"} Oct 31 04:14:45.593203 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.593181 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc" event={"ID":"7e70b0f0-c12d-4d2d-9a61-e059c63c071d","Type":"ContainerDied","Data":"359fc24f739ec3ea5a9fa462908b8adbc0aff230f46548c8f3037ad42c5feeb9"} Oct 31 04:14:45.593203 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.593203 5490 scope.go:122] "RemoveContainer" containerID="fb1f26d5d694a4b8d9640c213fdf14bc272b85ed90ab8829c64abd0e15b7f79b" Oct 31 04:14:45.603199 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.603176 5490 scope.go:122] "RemoveContainer" containerID="fb1f26d5d694a4b8d9640c213fdf14bc272b85ed90ab8829c64abd0e15b7f79b" Oct 31 04:14:45.603489 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:14:45.603470 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1f26d5d694a4b8d9640c213fdf14bc272b85ed90ab8829c64abd0e15b7f79b\": container with ID starting with fb1f26d5d694a4b8d9640c213fdf14bc272b85ed90ab8829c64abd0e15b7f79b not found: ID does not exist" containerID="fb1f26d5d694a4b8d9640c213fdf14bc272b85ed90ab8829c64abd0e15b7f79b" Oct 31 04:14:45.603551 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.603499 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1f26d5d694a4b8d9640c213fdf14bc272b85ed90ab8829c64abd0e15b7f79b"} err="failed to get container status \"fb1f26d5d694a4b8d9640c213fdf14bc272b85ed90ab8829c64abd0e15b7f79b\": rpc error: code = NotFound desc = could not find container \"fb1f26d5d694a4b8d9640c213fdf14bc272b85ed90ab8829c64abd0e15b7f79b\": container with ID starting with fb1f26d5d694a4b8d9640c213fdf14bc272b85ed90ab8829c64abd0e15b7f79b not found: ID does not exist" Oct 31 04:14:45.612280 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.612242 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc"] Oct 31 04:14:45.616078 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.616052 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-5e424f-6db94755f4-jddxc"] Oct 31 04:14:45.718115 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:45.718066 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:47.128178 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:14:47.128142 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:14:47.574513 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:47.574465 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e70b0f0-c12d-4d2d-9a61-e059c63c071d" path="/var/lib/kubelet/pods/7e70b0f0-c12d-4d2d-9a61-e059c63c071d/volumes" Oct 31 04:14:51.420866 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:51.420825 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:51.569552 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:51.569510 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:51.651941 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:51.651903 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424"] Oct 31 04:14:51.652207 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:51.652176 5490 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" podUID="ba6229f3-9eaa-4d9e-ac22-be70a9060eca" containerName="istio-proxy" containerID="cri-o://052b227570ab95e22bb84bbd89d478326bed9227401b782e6a3b5764fe8ac5d8" gracePeriod=30 Oct 31 04:14:51.787352 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:51.787259 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:14:55.953859 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:55.953803 5490 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-svnrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 04:14:55.954560 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:55.954520 5490 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-svnrd" podUID="b3c2974d-83ad-4321-93eb-60ec9738d588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 04:14:56.936293 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:56.936224 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:14:57.104133 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.104094 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-podinfo\") pod \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " Oct 31 04:14:57.104133 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.104138 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istiod-ca-cert\") pod \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " Oct 31 04:14:57.104749 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.104187 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-workload-certs\") pod \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " Oct 31 04:14:57.104749 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.104225 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-credential-socket\") pod \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " Oct 31 04:14:57.104749 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.104261 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-token\" (UniqueName: \"kubernetes.io/projected/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-token\") pod \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " Oct 31 04:14:57.104749 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.104291 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-envoy\") pod \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " Oct 31 04:14:57.104749 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.104330 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-workload-socket\") pod \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " Oct 31 04:14:57.104749 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.104388 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-data\") pod \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " Oct 31 04:14:57.104749 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.104443 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-kube-api-access-lsm6r\" (UniqueName: \"kubernetes.io/projected/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-kube-api-access-lsm6r\") pod \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\" (UID: \"ba6229f3-9eaa-4d9e-ac22-be70a9060eca\") " Oct 31 04:14:57.104749 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.104569 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-workload-certs" pod "ba6229f3-9eaa-4d9e-ac22-be70a9060eca" (UID: "ba6229f3-9eaa-4d9e-ac22-be70a9060eca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:57.105139 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.104656 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istiod-ca-cert" pod "ba6229f3-9eaa-4d9e-ac22-be70a9060eca" (UID: "ba6229f3-9eaa-4d9e-ac22-be70a9060eca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:14:57.105139 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.104844 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-credential-socket" pod "ba6229f3-9eaa-4d9e-ac22-be70a9060eca" (UID: "ba6229f3-9eaa-4d9e-ac22-be70a9060eca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:57.105139 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.104857 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-data" pod "ba6229f3-9eaa-4d9e-ac22-be70a9060eca" (UID: "ba6229f3-9eaa-4d9e-ac22-be70a9060eca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:57.105139 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.104777 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-workload-socket" pod "ba6229f3-9eaa-4d9e-ac22-be70a9060eca" (UID: "ba6229f3-9eaa-4d9e-ac22-be70a9060eca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:57.105139 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.104860 5490 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-workload-certs\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:14:57.106557 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.106531 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-token" pod "ba6229f3-9eaa-4d9e-ac22-be70a9060eca" (UID: "ba6229f3-9eaa-4d9e-ac22-be70a9060eca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:57.106766 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.106749 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-kube-api-access-lsm6r" pod "ba6229f3-9eaa-4d9e-ac22-be70a9060eca" (UID: "ba6229f3-9eaa-4d9e-ac22-be70a9060eca"). InnerVolumeSpecName "kube-api-access-lsm6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:14:57.106898 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.106880 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-podinfo" pod "ba6229f3-9eaa-4d9e-ac22-be70a9060eca" (UID: "ba6229f3-9eaa-4d9e-ac22-be70a9060eca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 04:14:57.106898 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.106886 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-envoy" pod "ba6229f3-9eaa-4d9e-ac22-be70a9060eca" (UID: "ba6229f3-9eaa-4d9e-ac22-be70a9060eca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 04:14:57.206201 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.206104 5490 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-podinfo\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:14:57.206201 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.206142 5490 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istiod-ca-cert\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:14:57.206201 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.206159 5490 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-credential-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:14:57.206201 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.206171 5490 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-token\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:14:57.206201 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.206183 5490 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-envoy\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:14:57.206201 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.206197 5490 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-workload-socket\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:14:57.206529 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.206210 5490 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-istio-data\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:14:57.206529 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.206223 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsm6r\" (UniqueName: \"kubernetes.io/projected/ba6229f3-9eaa-4d9e-ac22-be70a9060eca-kube-api-access-lsm6r\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:14:57.690591 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.690550 5490 generic.go:356] "Generic (PLEG): container finished" podID="ba6229f3-9eaa-4d9e-ac22-be70a9060eca" containerID="052b227570ab95e22bb84bbd89d478326bed9227401b782e6a3b5764fe8ac5d8" exitCode=0 Oct 31 04:14:57.690789 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.690600 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" Oct 31 04:14:57.690789 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.690642 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" event={"ID":"ba6229f3-9eaa-4d9e-ac22-be70a9060eca","Type":"ContainerDied","Data":"052b227570ab95e22bb84bbd89d478326bed9227401b782e6a3b5764fe8ac5d8"} Oct 31 04:14:57.690789 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.690709 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424" event={"ID":"ba6229f3-9eaa-4d9e-ac22-be70a9060eca","Type":"ContainerDied","Data":"0e6393c6e5ce04de15e9211502785bbe907788cfb5bb287af6d6444c44bcfe84"} Oct 31 04:14:57.690789 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.690732 5490 scope.go:122] "RemoveContainer" containerID="052b227570ab95e22bb84bbd89d478326bed9227401b782e6a3b5764fe8ac5d8" Oct 31 04:14:57.700365 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.700342 5490 scope.go:122] "RemoveContainer" containerID="252a43e73e28507eb9b5f19e9849ee4f2743776ff6dfe1043761ed2f5f29859e" Oct 31 04:14:57.709954 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.709930 5490 scope.go:122] "RemoveContainer" containerID="052b227570ab95e22bb84bbd89d478326bed9227401b782e6a3b5764fe8ac5d8" Oct 31 04:14:57.710721 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:14:57.710427 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"052b227570ab95e22bb84bbd89d478326bed9227401b782e6a3b5764fe8ac5d8\": container with ID starting with 052b227570ab95e22bb84bbd89d478326bed9227401b782e6a3b5764fe8ac5d8 not found: ID does not exist" containerID="052b227570ab95e22bb84bbd89d478326bed9227401b782e6a3b5764fe8ac5d8" Oct 31 04:14:57.710840 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.710737 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"052b227570ab95e22bb84bbd89d478326bed9227401b782e6a3b5764fe8ac5d8"} err="failed to get container status \"052b227570ab95e22bb84bbd89d478326bed9227401b782e6a3b5764fe8ac5d8\": rpc error: code = NotFound desc = could not find container \"052b227570ab95e22bb84bbd89d478326bed9227401b782e6a3b5764fe8ac5d8\": container with ID starting with 052b227570ab95e22bb84bbd89d478326bed9227401b782e6a3b5764fe8ac5d8 not found: ID does not exist" Oct 31 04:14:57.710840 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.710766 5490 scope.go:122] "RemoveContainer" containerID="252a43e73e28507eb9b5f19e9849ee4f2743776ff6dfe1043761ed2f5f29859e" Oct 31 04:14:57.711557 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:14:57.711532 5490 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"252a43e73e28507eb9b5f19e9849ee4f2743776ff6dfe1043761ed2f5f29859e\": container with ID starting with 252a43e73e28507eb9b5f19e9849ee4f2743776ff6dfe1043761ed2f5f29859e not found: ID does not exist" containerID="252a43e73e28507eb9b5f19e9849ee4f2743776ff6dfe1043761ed2f5f29859e" Oct 31 04:14:57.711663 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.711565 5490 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"252a43e73e28507eb9b5f19e9849ee4f2743776ff6dfe1043761ed2f5f29859e"} err="failed to get container status \"252a43e73e28507eb9b5f19e9849ee4f2743776ff6dfe1043761ed2f5f29859e\": rpc error: code = NotFound desc = could not find container \"252a43e73e28507eb9b5f19e9849ee4f2743776ff6dfe1043761ed2f5f29859e\": container with ID starting with 252a43e73e28507eb9b5f19e9849ee4f2743776ff6dfe1043761ed2f5f29859e not found: ID does not exist" Oct 31 04:14:57.713805 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.713767 5490 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424"] Oct 31 04:14:57.716275 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:57.716249 5490 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-5e424f-openshift-default-55f45c4fbf-xt424"] Oct 31 04:14:59.575000 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:14:59.574957 5490 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba6229f3-9eaa-4d9e-ac22-be70a9060eca" path="/var/lib/kubelet/pods/ba6229f3-9eaa-4d9e-ac22-be70a9060eca/volumes" Oct 31 04:15:03.293797 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.293760 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ms8vx"] Oct 31 04:15:03.294208 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.294195 5490 cpu_manager.go:414] "containerMap: removing container" podUID="ba6229f3-9eaa-4d9e-ac22-be70a9060eca" containerName="istio-proxy" Oct 31 04:15:03.294256 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.294210 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6229f3-9eaa-4d9e-ac22-be70a9060eca" containerName="istio-proxy" Oct 31 04:15:03.294256 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.294239 5490 cpu_manager.go:414] "containerMap: removing container" podUID="7e70b0f0-c12d-4d2d-9a61-e059c63c071d" containerName="maas-api" Oct 31 04:15:03.294256 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.294245 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e70b0f0-c12d-4d2d-9a61-e059c63c071d" containerName="maas-api" Oct 31 04:15:03.294256 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.294256 5490 cpu_manager.go:414] "containerMap: removing container" podUID="3d77f97d-a54d-4e24-a128-e15fa735e656" containerName="istio-proxy" Oct 31 04:15:03.294392 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.294261 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d77f97d-a54d-4e24-a128-e15fa735e656" containerName="istio-proxy" Oct 31 04:15:03.294392 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.294329 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d77f97d-a54d-4e24-a128-e15fa735e656" containerName="istio-proxy" Oct 31 04:15:03.294392 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.294340 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6229f3-9eaa-4d9e-ac22-be70a9060eca" containerName="istio-proxy" Oct 31 04:15:03.294392 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.294349 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e70b0f0-c12d-4d2d-9a61-e059c63c071d" containerName="maas-api" Oct 31 04:15:03.299139 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.299118 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ms8vx" Oct 31 04:15:03.302952 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.302928 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cq5w7\"" Oct 31 04:15:03.303097 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.302993 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 04:15:03.304916 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.304891 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ms8vx"] Oct 31 04:15:03.325338 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.325305 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-ms8vx\" (UID: \"20f1ffe0-5824-43f0-ad4f-94ce21843ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ms8vx" Oct 31 04:15:03.325465 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.325425 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-ms8vx\" (UID: \"20f1ffe0-5824-43f0-ad4f-94ce21843ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ms8vx" Oct 31 04:15:03.325513 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.325505 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqxmf\" (UniqueName: \"kubernetes.io/projected/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-kube-api-access-fqxmf\") pod \"maas-api-revoke-sub-f03629db-ms8vx\" (UID: \"20f1ffe0-5824-43f0-ad4f-94ce21843ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ms8vx" Oct 31 04:15:03.426857 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.426815 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-ms8vx\" (UID: \"20f1ffe0-5824-43f0-ad4f-94ce21843ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ms8vx" Oct 31 04:15:03.427037 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.426875 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-ms8vx\" (UID: \"20f1ffe0-5824-43f0-ad4f-94ce21843ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ms8vx" Oct 31 04:15:03.427037 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.426921 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqxmf\" (UniqueName: \"kubernetes.io/projected/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-kube-api-access-fqxmf\") pod \"maas-api-revoke-sub-f03629db-ms8vx\" (UID: \"20f1ffe0-5824-43f0-ad4f-94ce21843ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ms8vx" Oct 31 04:15:03.427630 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.427607 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-ms8vx\" (UID: \"20f1ffe0-5824-43f0-ad4f-94ce21843ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ms8vx" Oct 31 04:15:03.429359 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.429333 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-ms8vx\" (UID: \"20f1ffe0-5824-43f0-ad4f-94ce21843ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ms8vx" Oct 31 04:15:03.434294 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.434276 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqxmf\" (UniqueName: \"kubernetes.io/projected/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-kube-api-access-fqxmf\") pod \"maas-api-revoke-sub-f03629db-ms8vx\" (UID: \"20f1ffe0-5824-43f0-ad4f-94ce21843ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ms8vx" Oct 31 04:15:03.615718 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.615605 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ms8vx" Oct 31 04:15:03.890568 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.890541 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ms8vx"] Oct 31 04:15:03.893067 ip-10-0-142-76 kubenswrapper[5490]: W1031 04:15:03.893034 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f1ffe0_5824_43f0_ad4f_94ce21843ddc.slice/crio-b5c7463ba2e56f0b6e7178f25792afd315be4a11f3c1642d9e56ae9f72ee892c WatchSource:0}: Error finding container b5c7463ba2e56f0b6e7178f25792afd315be4a11f3c1642d9e56ae9f72ee892c: Status 404 returned error can't find the container with id b5c7463ba2e56f0b6e7178f25792afd315be4a11f3c1642d9e56ae9f72ee892c Oct 31 04:15:03.895075 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:03.895056 5490 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 04:15:04.733053 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:04.733013 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ms8vx" event={"ID":"20f1ffe0-5824-43f0-ad4f-94ce21843ddc","Type":"ContainerStarted","Data":"b5c7463ba2e56f0b6e7178f25792afd315be4a11f3c1642d9e56ae9f72ee892c"} Oct 31 04:15:07.752867 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:07.752829 5490 generic.go:356] "Generic (PLEG): container finished" podID="20f1ffe0-5824-43f0-ad4f-94ce21843ddc" containerID="d25217f9ad51209130e48b70f3675f3db108f98b2f76bf02cd6c9f9ab074aff5" exitCode=0 Oct 31 04:15:07.753295 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:07.752904 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ms8vx" event={"ID":"20f1ffe0-5824-43f0-ad4f-94ce21843ddc","Type":"ContainerDied","Data":"d25217f9ad51209130e48b70f3675f3db108f98b2f76bf02cd6c9f9ab074aff5"} Oct 31 04:15:08.911006 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:08.910977 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ms8vx" Oct 31 04:15:08.983619 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:08.983582 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-maas-api-xks-ca\") pod \"20f1ffe0-5824-43f0-ad4f-94ce21843ddc\" (UID: \"20f1ffe0-5824-43f0-ad4f-94ce21843ddc\") " Oct 31 04:15:08.983856 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:08.983659 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-kube-api-access-fqxmf\" (UniqueName: \"kubernetes.io/projected/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-kube-api-access-fqxmf\") pod \"20f1ffe0-5824-43f0-ad4f-94ce21843ddc\" (UID: \"20f1ffe0-5824-43f0-ad4f-94ce21843ddc\") " Oct 31 04:15:08.983856 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:08.983747 5490 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-maas-api-openshift-ca\") pod \"20f1ffe0-5824-43f0-ad4f-94ce21843ddc\" (UID: \"20f1ffe0-5824-43f0-ad4f-94ce21843ddc\") " Oct 31 04:15:08.984154 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:08.984129 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-maas-api-openshift-ca" pod "20f1ffe0-5824-43f0-ad4f-94ce21843ddc" (UID: "20f1ffe0-5824-43f0-ad4f-94ce21843ddc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 04:15:08.985916 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:08.985888 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-kube-api-access-fqxmf" pod "20f1ffe0-5824-43f0-ad4f-94ce21843ddc" (UID: "20f1ffe0-5824-43f0-ad4f-94ce21843ddc"). InnerVolumeSpecName "kube-api-access-fqxmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 04:15:08.986025 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:08.985931 5490 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-maas-api-xks-ca" pod "20f1ffe0-5824-43f0-ad4f-94ce21843ddc" (UID: "20f1ffe0-5824-43f0-ad4f-94ce21843ddc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 04:15:09.084614 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:09.084524 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-maas-api-xks-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:15:09.084614 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:09.084561 5490 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqxmf\" (UniqueName: \"kubernetes.io/projected/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-kube-api-access-fqxmf\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:15:09.084614 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:09.084575 5490 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20f1ffe0-5824-43f0-ad4f-94ce21843ddc-maas-api-openshift-ca\") on node \"ip-10-0-142-76.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 04:15:09.765738 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:09.765702 5490 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ms8vx" Oct 31 04:15:09.765738 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:09.765721 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-ms8vx" event={"ID":"20f1ffe0-5824-43f0-ad4f-94ce21843ddc","Type":"ContainerDied","Data":"b5c7463ba2e56f0b6e7178f25792afd315be4a11f3c1642d9e56ae9f72ee892c"} Oct 31 04:15:09.765962 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:09.765771 5490 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5c7463ba2e56f0b6e7178f25792afd315be4a11f3c1642d9e56ae9f72ee892c" Oct 31 04:15:17.140097 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:15:17.140043 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:15:19.266246 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:19.266200 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:41.196547 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:41.196513 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b8d5bdb45-vldp7_c8be91e5-ce63-477c-91c3-4a24e694a0f3/authorino/0.log" Oct 31 04:15:41.759276 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:41.759236 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:41.907347 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:41.906882 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:15:47.127889 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:15:47.127848 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:15:51.393187 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:51.393147 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-wnnv7_ce529bc9-a44d-4808-8d2c-996e14f88a54/manager/1.log" Oct 31 04:15:51.569907 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:51.569873 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wlb6z_30874c80-e957-4c9a-ab8f-0616de567c3d/manager/1.log" Oct 31 04:15:52.066239 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:52.066196 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-586ddd746f-hdwdt_f21a691b-8bb1-480d-8d90-d6b78aac8624/manager/0.log" Oct 31 04:15:52.415963 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:52.415929 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-s9sn6_ce9fae9e-74e8-4406-bc98-fd4ba1a319bd/manager/0.log" Oct 31 04:15:52.618322 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:52.618284 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-68drb_55f29950-78b4-4a52-818d-dc235e4631fc/proxy/1.log" Oct 31 04:15:52.966052 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:52.966021 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-wnnv7_ce529bc9-a44d-4808-8d2c-996e14f88a54/manager/0.log" Oct 31 04:15:53.162717 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:53.162658 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wlb6z_30874c80-e957-4c9a-ab8f-0616de567c3d/manager/0.log" Oct 31 04:15:53.364275 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:53.364195 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-68drb_55f29950-78b4-4a52-818d-dc235e4631fc/proxy/0.log" Oct 31 04:15:54.049627 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:54.049598 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-bf8887774-hr8xq_7cb9f5f1-55a1-4702-a299-67cae7e36032/maas-api/0.log" Oct 31 04:15:55.096846 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:55.096815 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-ms8vx_20f1ffe0-5824-43f0-ad4f-94ce21843ddc/revoke-keys/0.log" Oct 31 04:15:55.269242 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:55.269209 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-82ctb_85995108-6b7b-4e56-8260-00b5347d6e6a/postgres/0.log" Oct 31 04:15:56.720530 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:56.720496 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b8d5bdb45-vldp7_c8be91e5-ce63-477c-91c3-4a24e694a0f3/authorino/0.log" Oct 31 04:15:57.845366 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:57.845327 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-f2tk8_efd854cd-40a6-4f9f-bd19-cd5d19512dba/manager/1.log" Oct 31 04:15:58.256530 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:58.256494 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-f2tk8_efd854cd-40a6-4f9f-bd19-cd5d19512dba/manager/0.log" Oct 31 04:15:59.153050 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:15:59.153011 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86bb8bf66f-tqrbj_01171a38-57a8-46bb-b2a0-9d69f73f12ea/kube-auth-proxy/0.log" Oct 31 04:16:00.570892 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:00.570849 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-rssq8_8eae7b8f-98ca-4de7-92e0-ff47b28d6927/istio-proxy/0.log" Oct 31 04:16:00.750184 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:00.750149 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-svnrd_b3c2974d-83ad-4321-93eb-60ec9738d588/istio-proxy/0.log" Oct 31 04:16:00.886913 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:00.886830 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:00.911076 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:00.911039 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-t9bq7_6ffcf283-0d79-422e-82be-8b34db814efd/istio-proxy/0.log" Oct 31 04:16:01.420897 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:01.420861 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-gzjbq_c9d776b1-fabe-4396-adb1-030d1edad90c/payload-pre-processing/0.log" Oct 31 04:16:01.582830 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:01.582795 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rclxh_9e7f6e11-0382-4897-88e3-1d4283f5bed3/payload-processing/0.log" Oct 31 04:16:02.518348 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:02.518305 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8_775a8940-cb22-4774-a43f-c6e7c5364354/main/0.log" Oct 31 04:16:03.014950 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:03.014913 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-b78nf_80180652-fb4f-4976-956d-25a47344d581/main/0.log" Oct 31 04:16:03.335413 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:03.335303 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-vczbq_e9ecffa6-8de1-421d-952e-a9eeecf8d862/main/0.log" Oct 31 04:16:03.496527 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:03.496490 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm_f64d3e84-440a-4298-9230-ae9e0063c962/main/0.log" Oct 31 04:16:04.527997 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:04.527942 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b8d5bdb45-vldp7_c8be91e5-ce63-477c-91c3-4a24e694a0f3/authorino/0.log" Oct 31 04:16:13.388876 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:13.388777 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-wnnv7_ce529bc9-a44d-4808-8d2c-996e14f88a54/manager/1.log" Oct 31 04:16:13.575273 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:13.575239 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wlb6z_30874c80-e957-4c9a-ab8f-0616de567c3d/manager/1.log" Oct 31 04:16:14.006493 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:14.006454 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-586ddd746f-hdwdt_f21a691b-8bb1-480d-8d90-d6b78aac8624/manager/0.log" Oct 31 04:16:14.411179 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:14.411146 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-s9sn6_ce9fae9e-74e8-4406-bc98-fd4ba1a319bd/manager/0.log" Oct 31 04:16:14.609566 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:14.609529 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-68drb_55f29950-78b4-4a52-818d-dc235e4631fc/proxy/1.log" Oct 31 04:16:15.313671 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:15.313636 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-bf8887774-hr8xq_7cb9f5f1-55a1-4702-a299-67cae7e36032/maas-api/0.log" Oct 31 04:16:16.352945 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:16.352763 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-ms8vx_20f1ffe0-5824-43f0-ad4f-94ce21843ddc/revoke-keys/0.log" Oct 31 04:16:16.514496 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:16.514455 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-82ctb_85995108-6b7b-4e56-8260-00b5347d6e6a/postgres/0.log" Oct 31 04:16:17.127004 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:16:17.126965 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:16:17.640845 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:17.640807 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b8d5bdb45-vldp7_c8be91e5-ce63-477c-91c3-4a24e694a0f3/authorino/0.log" Oct 31 04:16:18.815196 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:18.815159 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-f2tk8_efd854cd-40a6-4f9f-bd19-cd5d19512dba/manager/1.log" Oct 31 04:16:19.306811 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:19.306768 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:16:19.733841 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:19.733801 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86bb8bf66f-tqrbj_01171a38-57a8-46bb-b2a0-9d69f73f12ea/kube-auth-proxy/0.log" Oct 31 04:16:21.196846 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:21.196809 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-rssq8_8eae7b8f-98ca-4de7-92e0-ff47b28d6927/istio-proxy/0.log" Oct 31 04:16:21.381536 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:21.381471 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-svnrd_b3c2974d-83ad-4321-93eb-60ec9738d588/istio-proxy/0.log" Oct 31 04:16:21.569383 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:21.569269 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-t9bq7_6ffcf283-0d79-422e-82be-8b34db814efd/istio-proxy/0.log" Oct 31 04:16:22.081854 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:22.081825 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-gzjbq_c9d776b1-fabe-4396-adb1-030d1edad90c/payload-pre-processing/0.log" Oct 31 04:16:22.254173 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:22.254140 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rclxh_9e7f6e11-0382-4897-88e3-1d4283f5bed3/payload-processing/0.log" Oct 31 04:16:22.930539 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:22.930505 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-gwwr8_775a8940-cb22-4774-a43f-c6e7c5364354/main/0.log" Oct 31 04:16:23.461159 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:23.461125 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-b78nf_80180652-fb4f-4976-956d-25a47344d581/main/0.log" Oct 31 04:16:23.803461 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:23.803380 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-vczbq_e9ecffa6-8de1-421d-952e-a9eeecf8d862/main/0.log" Oct 31 04:16:23.979779 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:16:23.979746 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-gxzfm_f64d3e84-440a-4298-9230-ae9e0063c962/main/0.log" Oct 31 04:16:47.127024 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:16:47.126988 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:17:01.234990 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:17:01.234948 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-586ddd746f-hdwdt_f21a691b-8bb1-480d-8d90-d6b78aac8624/manager/0.log" Oct 31 04:17:01.492298 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:17:01.492205 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-586ddd746f-hdwdt_f21a691b-8bb1-480d-8d90-d6b78aac8624/manager/0.log" Oct 31 04:17:02.129522 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:17:02.129494 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rclxh_9e7f6e11-0382-4897-88e3-1d4283f5bed3/payload-processing/0.log" Oct 31 04:17:02.310908 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:17:02.310880 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rclxh_9e7f6e11-0382-4897-88e3-1d4283f5bed3/payload-processing/0.log" Oct 31 04:17:12.065936 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:17:12.065893 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:17:17.127540 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:17:17.127499 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:17:18.732985 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:17:18.732941 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:17:18.865357 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:17:18.865319 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:17:19.350976 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:17:19.350934 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:17:47.127078 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:17:47.126994 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:17:52.798666 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:17:52.798621 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-586ddd746f-hdwdt_f21a691b-8bb1-480d-8d90-d6b78aac8624/manager/0.log" Oct 31 04:17:53.066739 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:17:53.066645 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-586ddd746f-hdwdt_f21a691b-8bb1-480d-8d90-d6b78aac8624/manager/0.log" Oct 31 04:17:53.778929 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:17:53.778896 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rclxh_9e7f6e11-0382-4897-88e3-1d4283f5bed3/payload-processing/0.log" Oct 31 04:17:53.955479 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:17:53.955453 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rclxh_9e7f6e11-0382-4897-88e3-1d4283f5bed3/payload-processing/0.log" Oct 31 04:17:54.309170 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:17:54.309132 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:17:54.566915 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:17:54.566816 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:17:57.434431 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:17:57.434393 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gkf5m_7d877fe5-7b94-4645-bff5-6df0796bb637/global-pull-secret-syncer/0.log" Oct 31 04:17:57.580521 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:17:57.580468 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" Oct 31 04:17:57.777869 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:17:57.777770 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-76.us-east-2.compute.internal_fc792e750c2606b177de944a501bf339/haproxy/0.log" Oct 31 04:18:04.924072 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:04.924032 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b8d5bdb45-vldp7_c8be91e5-ce63-477c-91c3-4a24e694a0f3/authorino/0.log" Oct 31 04:18:05.236312 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:05.236217 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-f2tk8_efd854cd-40a6-4f9f-bd19-cd5d19512dba/manager/0.log" Oct 31 04:18:05.244327 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:05.244295 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-f2tk8_efd854cd-40a6-4f9f-bd19-cd5d19512dba/manager/1.log" Oct 31 04:18:06.863688 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:06.863651 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-5kbd8_8bae2714-9c1b-44db-a6ef-b0b008cf1fa2/monitoring-plugin/0.log" Oct 31 04:18:07.007893 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.007852 5490 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw"] Oct 31 04:18:07.008477 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.008448 5490 cpu_manager.go:414] "containerMap: removing container" podUID="ba6229f3-9eaa-4d9e-ac22-be70a9060eca" containerName="istio-proxy" Oct 31 04:18:07.008477 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.008471 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6229f3-9eaa-4d9e-ac22-be70a9060eca" containerName="istio-proxy" Oct 31 04:18:07.008640 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.008525 5490 cpu_manager.go:414] "containerMap: removing container" podUID="20f1ffe0-5824-43f0-ad4f-94ce21843ddc" containerName="revoke-keys" Oct 31 04:18:07.008640 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.008533 5490 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f1ffe0-5824-43f0-ad4f-94ce21843ddc" containerName="revoke-keys" Oct 31 04:18:07.008640 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.008597 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f1ffe0-5824-43f0-ad4f-94ce21843ddc" containerName="revoke-keys" Oct 31 04:18:07.008640 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.008608 5490 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6229f3-9eaa-4d9e-ac22-be70a9060eca" containerName="istio-proxy" Oct 31 04:18:07.011840 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.011821 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" Oct 31 04:18:07.019768 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.019417 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw"] Oct 31 04:18:07.078496 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.078457 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mtd46_575aa82a-6d21-46c1-80ff-7f756f00c108/node-exporter/0.log" Oct 31 04:18:07.102064 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.102036 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mtd46_575aa82a-6d21-46c1-80ff-7f756f00c108/kube-rbac-proxy/0.log" Oct 31 04:18:07.104947 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.104924 5490 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6qbd4\"/\"default-dockercfg-f7krs\"" Oct 31 04:18:07.107832 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.107806 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6qbd4\"/\"openshift-service-ca.crt\"" Oct 31 04:18:07.107832 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.107812 5490 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6qbd4\"/\"kube-root-ca.crt\"" Oct 31 04:18:07.125011 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.124945 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mtd46_575aa82a-6d21-46c1-80ff-7f756f00c108/init-textfile/0.log" Oct 31 04:18:07.152413 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.152378 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f520d16b-c9a1-4a9c-8ae8-de8afbf94964-lib-modules\") pod \"perf-node-gather-daemonset-cvgcw\" (UID: \"f520d16b-c9a1-4a9c-8ae8-de8afbf94964\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" Oct 31 04:18:07.152588 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.152449 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f520d16b-c9a1-4a9c-8ae8-de8afbf94964-podres\") pod \"perf-node-gather-daemonset-cvgcw\" (UID: \"f520d16b-c9a1-4a9c-8ae8-de8afbf94964\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" Oct 31 04:18:07.152588 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.152484 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2t5m\" (UniqueName: \"kubernetes.io/projected/f520d16b-c9a1-4a9c-8ae8-de8afbf94964-kube-api-access-q2t5m\") pod \"perf-node-gather-daemonset-cvgcw\" (UID: \"f520d16b-c9a1-4a9c-8ae8-de8afbf94964\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" Oct 31 04:18:07.152588 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.152542 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f520d16b-c9a1-4a9c-8ae8-de8afbf94964-sys\") pod \"perf-node-gather-daemonset-cvgcw\" (UID: \"f520d16b-c9a1-4a9c-8ae8-de8afbf94964\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" Oct 31 04:18:07.152588 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.152577 5490 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f520d16b-c9a1-4a9c-8ae8-de8afbf94964-proc\") pod \"perf-node-gather-daemonset-cvgcw\" (UID: \"f520d16b-c9a1-4a9c-8ae8-de8afbf94964\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" Oct 31 04:18:07.239609 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.239568 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c835066f-39c8-42b4-9f99-1055495bcc69/prometheus/0.log" Oct 31 04:18:07.253878 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.253840 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f520d16b-c9a1-4a9c-8ae8-de8afbf94964-lib-modules\") pod \"perf-node-gather-daemonset-cvgcw\" (UID: \"f520d16b-c9a1-4a9c-8ae8-de8afbf94964\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" Oct 31 04:18:07.254076 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.253921 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f520d16b-c9a1-4a9c-8ae8-de8afbf94964-podres\") pod \"perf-node-gather-daemonset-cvgcw\" (UID: \"f520d16b-c9a1-4a9c-8ae8-de8afbf94964\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" Oct 31 04:18:07.254076 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.253945 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2t5m\" (UniqueName: \"kubernetes.io/projected/f520d16b-c9a1-4a9c-8ae8-de8afbf94964-kube-api-access-q2t5m\") pod \"perf-node-gather-daemonset-cvgcw\" (UID: \"f520d16b-c9a1-4a9c-8ae8-de8afbf94964\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" Oct 31 04:18:07.254076 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.253983 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f520d16b-c9a1-4a9c-8ae8-de8afbf94964-sys\") pod \"perf-node-gather-daemonset-cvgcw\" (UID: \"f520d16b-c9a1-4a9c-8ae8-de8afbf94964\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" Oct 31 04:18:07.254076 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.253998 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f520d16b-c9a1-4a9c-8ae8-de8afbf94964-lib-modules\") pod \"perf-node-gather-daemonset-cvgcw\" (UID: \"f520d16b-c9a1-4a9c-8ae8-de8afbf94964\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" Oct 31 04:18:07.254076 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.254035 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f520d16b-c9a1-4a9c-8ae8-de8afbf94964-sys\") pod \"perf-node-gather-daemonset-cvgcw\" (UID: \"f520d16b-c9a1-4a9c-8ae8-de8afbf94964\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" Oct 31 04:18:07.254076 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.254046 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f520d16b-c9a1-4a9c-8ae8-de8afbf94964-podres\") pod \"perf-node-gather-daemonset-cvgcw\" (UID: \"f520d16b-c9a1-4a9c-8ae8-de8afbf94964\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" Oct 31 04:18:07.254334 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.254131 5490 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f520d16b-c9a1-4a9c-8ae8-de8afbf94964-proc\") pod \"perf-node-gather-daemonset-cvgcw\" (UID: \"f520d16b-c9a1-4a9c-8ae8-de8afbf94964\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" Oct 31 04:18:07.254334 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.254234 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f520d16b-c9a1-4a9c-8ae8-de8afbf94964-proc\") pod \"perf-node-gather-daemonset-cvgcw\" (UID: \"f520d16b-c9a1-4a9c-8ae8-de8afbf94964\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" Oct 31 04:18:07.262369 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.262329 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c835066f-39c8-42b4-9f99-1055495bcc69/config-reloader/0.log" Oct 31 04:18:07.262543 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.262517 5490 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2t5m\" (UniqueName: \"kubernetes.io/projected/f520d16b-c9a1-4a9c-8ae8-de8afbf94964-kube-api-access-q2t5m\") pod \"perf-node-gather-daemonset-cvgcw\" (UID: \"f520d16b-c9a1-4a9c-8ae8-de8afbf94964\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" Oct 31 04:18:07.285843 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.285793 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c835066f-39c8-42b4-9f99-1055495bcc69/thanos-sidecar/0.log" Oct 31 04:18:07.309711 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.309666 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c835066f-39c8-42b4-9f99-1055495bcc69/kube-rbac-proxy-web/0.log" Oct 31 04:18:07.326626 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.326588 5490 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" Oct 31 04:18:07.332666 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.332638 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c835066f-39c8-42b4-9f99-1055495bcc69/kube-rbac-proxy/0.log" Oct 31 04:18:07.355972 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.355939 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c835066f-39c8-42b4-9f99-1055495bcc69/kube-rbac-proxy-thanos/0.log" Oct 31 04:18:07.383527 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.383379 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c835066f-39c8-42b4-9f99-1055495bcc69/init-config-reloader/0.log" Oct 31 04:18:07.427323 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.427290 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-968x8_cf66bb31-fee6-49fd-aaea-c8e8d8361c77/prometheus-operator/0.log" Oct 31 04:18:07.444459 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.444426 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-968x8_cf66bb31-fee6-49fd-aaea-c8e8d8361c77/kube-rbac-proxy/0.log" Oct 31 04:18:07.591463 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.591434 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b949d9bf6-8b6xc_d395aca5-b8ff-4da1-ad31-dd26e2427c2e/thanos-query/0.log" Oct 31 04:18:07.607003 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.606978 5490 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw"] Oct 31 04:18:07.609170 ip-10-0-142-76 kubenswrapper[5490]: W1031 04:18:07.609140 5490 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf520d16b_c9a1_4a9c_8ae8_de8afbf94964.slice/crio-11400699d7d026e4522aaa4b625faedd40db81eb472ef365112a4d6fee424e9a WatchSource:0}: Error finding container 11400699d7d026e4522aaa4b625faedd40db81eb472ef365112a4d6fee424e9a: Status 404 returned error can't find the container with id 11400699d7d026e4522aaa4b625faedd40db81eb472ef365112a4d6fee424e9a Oct 31 04:18:07.617131 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.617099 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b949d9bf6-8b6xc_d395aca5-b8ff-4da1-ad31-dd26e2427c2e/kube-rbac-proxy-web/1.log" Oct 31 04:18:07.617615 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.617590 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b949d9bf6-8b6xc_d395aca5-b8ff-4da1-ad31-dd26e2427c2e/kube-rbac-proxy-web/0.log" Oct 31 04:18:07.637148 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.637074 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b949d9bf6-8b6xc_d395aca5-b8ff-4da1-ad31-dd26e2427c2e/kube-rbac-proxy/0.log" Oct 31 04:18:07.660122 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.660092 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b949d9bf6-8b6xc_d395aca5-b8ff-4da1-ad31-dd26e2427c2e/prom-label-proxy/0.log" Oct 31 04:18:07.686346 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.686283 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b949d9bf6-8b6xc_d395aca5-b8ff-4da1-ad31-dd26e2427c2e/kube-rbac-proxy-rules/0.log" Oct 31 04:18:07.710216 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.710187 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-b949d9bf6-8b6xc_d395aca5-b8ff-4da1-ad31-dd26e2427c2e/kube-rbac-proxy-metrics/0.log" Oct 31 04:18:07.764046 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.764007 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" event={"ID":"f520d16b-c9a1-4a9c-8ae8-de8afbf94964","Type":"ContainerStarted","Data":"63eb5f96d1be69878165305b1f8e4f6fb37a45605026f8b8b92c23ae251ea5dd"} Oct 31 04:18:07.764229 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.764069 5490 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" event={"ID":"f520d16b-c9a1-4a9c-8ae8-de8afbf94964","Type":"ContainerStarted","Data":"11400699d7d026e4522aaa4b625faedd40db81eb472ef365112a4d6fee424e9a"} Oct 31 04:18:07.764229 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.764196 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" Oct 31 04:18:07.779588 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:07.779539 5490 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" podStartSLOduration=1.779525809 podStartE2EDuration="1.779525809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:07.777878008 +0000 UTC m=+3722.312572965" watchObservedRunningTime="2026-10-31 04:18:07.779525809 +0000 UTC m=+3722.314220763" Oct 31 04:18:11.764198 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:11.764156 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vcg5q_062e4d3d-4460-4412-93d3-22681a9a74e6/dns/0.log" Oct 31 04:18:11.783526 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:11.783494 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vcg5q_062e4d3d-4460-4412-93d3-22681a9a74e6/kube-rbac-proxy/0.log" Oct 31 04:18:11.857530 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:11.857502 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z74z2_16b948f8-e3da-4285-8342-27cbd2dbd141/dns-node-resolver/0.log" Oct 31 04:18:12.437180 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:12.437150 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g2mdn_55f1a18a-59ef-4146-a0e8-3fc632ab20f5/node-ca/0.log" Oct 31 04:18:13.782920 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:13.782884 5490 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-cvgcw" Oct 31 04:18:17.009130 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:17.009097 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86bb8bf66f-tqrbj_01171a38-57a8-46bb-b2a0-9d69f73f12ea/kube-auth-proxy/0.log" Oct 31 04:18:17.128054 ip-10-0-142-76 kubenswrapper[5490]: E1031 04:18:17.128013 5490 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zngp8" containerName="extractor" Oct 31 04:18:17.280851 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:17.280763 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-rssq8_8eae7b8f-98ca-4de7-92e0-ff47b28d6927/istio-proxy/0.log" Oct 31 04:18:17.309212 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:17.309182 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-svnrd_b3c2974d-83ad-4321-93eb-60ec9738d588/istio-proxy/0.log" Oct 31 04:18:17.337822 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:17.337762 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-t9bq7_6ffcf283-0d79-422e-82be-8b34db814efd/istio-proxy/0.log" Oct 31 04:18:17.421253 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:17.421216 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-gzjbq_c9d776b1-fabe-4396-adb1-030d1edad90c/payload-pre-processing/0.log" Oct 31 04:18:17.447566 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:17.447535 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rclxh_9e7f6e11-0382-4897-88e3-1d4283f5bed3/payload-processing/0.log" Oct 31 04:18:18.478622 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:18.478594 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ttmkb_400b676a-a3a0-459d-9da6-8d30b2d3b497/csi-driver/0.log" Oct 31 04:18:19.473935 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:19.473895 5490 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5lds2_8f247bb3-6322-43c8-bfad-f27a41e26500/konnectivity-agent/0.log" limit=1024 Oct 31 04:18:20.362654 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:20.362621 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vh2gt_50818995-4b65-457e-9b55-250b548e80d3/serve-healthcheck-canary/0.log" Oct 31 04:18:20.778825 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:20.778792 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86bb8bf66f-tqrbj_01171a38-57a8-46bb-b2a0-9d69f73f12ea/kube-auth-proxy/0.log" Oct 31 04:18:20.928029 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:20.928001 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-rssq8_8eae7b8f-98ca-4de7-92e0-ff47b28d6927/istio-proxy/0.log" Oct 31 04:18:20.944041 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:20.944007 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-svnrd_b3c2974d-83ad-4321-93eb-60ec9738d588/istio-proxy/0.log" Oct 31 04:18:20.959935 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:20.959902 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-t9bq7_6ffcf283-0d79-422e-82be-8b34db814efd/istio-proxy/0.log" Oct 31 04:18:21.006393 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:21.006357 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-gzjbq_c9d776b1-fabe-4396-adb1-030d1edad90c/payload-pre-processing/0.log" Oct 31 04:18:21.020059 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:21.020030 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-rclxh_9e7f6e11-0382-4897-88e3-1d4283f5bed3/payload-processing/0.log" Oct 31 04:18:21.065964 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:21.065884 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zngp8_08da34c7-c69a-4eb9-848a-399cab14840e/kube-rbac-proxy/0.log" Oct 31 04:18:21.088127 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:21.088097 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zngp8_08da34c7-c69a-4eb9-848a-399cab14840e/exporter/0.log" Oct 31 04:18:21.112732 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:21.112704 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zngp8_08da34c7-c69a-4eb9-848a-399cab14840e/extractor/1.log" Oct 31 04:18:21.112925 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:21.112903 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zngp8_08da34c7-c69a-4eb9-848a-399cab14840e/extractor/0.log" Oct 31 04:18:23.626815 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:23.626780 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-55975df5cc-vtdtl_fd89b915-fe3a-4a8a-aaf5-6927987b2e0d/manager/1.log" Oct 31 04:18:23.659961 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:23.659927 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-55975df5cc-vtdtl_fd89b915-fe3a-4a8a-aaf5-6927987b2e0d/manager/0.log" Oct 31 04:18:23.679086 ip-10-0-142-76 kubenswrapper[5490]: I1031 04:18:23.679056 5490 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-55975df5cc-vtdtl_fd89b915-fe3a-4a8a-aaf5-6927987b2e0d/copy-manifests/0.log"